Nov 03 21:14:15.029652 ip-10-0-129-65 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 21:14:15.443740 ip-10-0-129-65 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:15.443740 ip-10-0-129-65 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 21:14:15.443740 ip-10-0-129-65 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:15.443740 ip-10-0-129-65 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:14:15.448654 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448566    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.448654 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448653    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.448654 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448657    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.448654 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448660    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448663    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448666    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448671    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448675    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448678    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448681    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448683    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448687    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448689    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448708    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448710    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448713    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448715    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448718    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448721    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448724    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448726    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448729    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448731    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.448771 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448734    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448737    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448739    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448742    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448744    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448746    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448749    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448751    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448754    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448756    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448759    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448762    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448764    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448767    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448770    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448773    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448776    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448779    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448781    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.449263 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448784    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448786    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448789    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448792    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448795    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448797    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448800    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448802    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448805    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448807    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448809    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448812    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448814    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448817    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448819    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448823    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448825    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448827    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448830    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448832    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.449719 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448835    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448837    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448840    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448842    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448845    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448847    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448849    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448853    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448856    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448860    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448864    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448867    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448870    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448872    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448874    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448877    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448879    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448882    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448885    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.450226 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448888    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448890    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448893    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448895    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448898    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448900    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448903    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448905    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448908    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448910    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448913    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448915    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448918    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448921    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448923    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448926    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448928    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448930    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448933    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448935    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.450706 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448939    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448941    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448945    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448948    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.448950    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449874    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449882    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449885    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449888    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449890    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449893    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449901    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449904    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449907    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449910    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449912    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449915    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449917    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449919    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.451273 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449922    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449925    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449927    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449929    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449932    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449935    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449937    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449940    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449942    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449944    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449947    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449949    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449952    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449958    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449961    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449963    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449966    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449968    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449971    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449973    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.451758 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449976    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449978    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449981    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449983    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449985    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449993    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449996    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.449999    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450002    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450004    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450006    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450009    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450011    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450014    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450016    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450020    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450024    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450027    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450029    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.452231 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450032    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450035    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450040    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450044    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450047    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450050    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450054    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450057    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450059    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450062    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450065    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450067    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450070    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450072    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450075    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450077    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450080    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450082    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450084    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450092    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.452712 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450095    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450098    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450100    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450103    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450105    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450108    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450110    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450112    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450115    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450117    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450119    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450122    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450124    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450126    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450129    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450132    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450134    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450136    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450141    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.453218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450144    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450146    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450149    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450151    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450154    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450156    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450167    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450171    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450173    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450176    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450178    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450181    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450184    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450192    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450773    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450779    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450782    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450785    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450788    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450790    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.453684 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450793    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450795    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450802    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450805    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450809    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450812    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450814    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450817    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450819    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450822    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450825    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450828    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450830    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450833    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450835    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450838    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450840    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450843    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450845    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.454157 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450848    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450850    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450852    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450855    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450857    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450860    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450863    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450866    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450868    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450871    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450873    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450876    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450878    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450881    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450883    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450889    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450891    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450894    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450896    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.454634 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450899    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450902    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450905    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450908    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450911    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450914    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450916    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450919    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450921    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450924    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450927    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450929    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450931    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450934    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450936    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450939    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450941    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450944    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450946    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.455091 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450948    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450957    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450960    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450963    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450965    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450967    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450970    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450972    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450975    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450980    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450983    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450985    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450988    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450991    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450993    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450995    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.450998    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451000    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451003    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.455547 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451006    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451008    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451011    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451013    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451016    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451018    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451020    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451023    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451025    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451028    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451030    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451033    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451035    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451038    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451040    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451049    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451051    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451054    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451056    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451060    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.455988 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451062    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451064    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451068    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.451071    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452151    2568 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452161    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452172    2568 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452178    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452182    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452185    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452190    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452194    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452197    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452200    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452204    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452207    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452210    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452213    2568 flags.go:64] FLAG: --cgroup-root=""
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452216    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452219    2568 flags.go:64] FLAG: --client-ca-file=""
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452222    2568 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452225    2568 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452231    2568 flags.go:64] FLAG: --cluster-domain=""
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452235    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 21:14:15.456482 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452238    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452241    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452244    2568 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452248    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452257    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452261    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452268    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452271    2568 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452275    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452277    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452281    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452284    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452288    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452291    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452294    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452297    2568 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452300    2568 flags.go:64] FLAG: --enable-server="true"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452303    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452310    2568 flags.go:64] FLAG: --event-burst="100"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452315    2568 flags.go:64] FLAG: --event-qps="50"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452318    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452321    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452336    2568 flags.go:64] FLAG: --eviction-hard=""
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452341    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 21:14:15.457041 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452344    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452347    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452350    2568 flags.go:64] FLAG: --eviction-soft=""
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452353    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452356    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452359    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452361    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452364    2568 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452367    2568 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452370    2568 flags.go:64] FLAG: --feature-gates=""
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452374    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452378    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452381    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452392    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452396    2568 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452399    2568 flags.go:64] FLAG: --help="false"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452402    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452405    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452408    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452411    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452415    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452418    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452421    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 21:14:15.457644 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452424    2568 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452427    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452429    2568 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452432    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452437    2568 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452439    2568 flags.go:64] FLAG: --kube-reserved=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452442    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452445    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452448    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452451    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452454    2568 flags.go:64] FLAG: --lock-file=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452456    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452459    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452462    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452468    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452471    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452474    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452477    2568 flags.go:64] FLAG: --logging-format="text"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452479    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452482    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452485    2568 flags.go:64] FLAG: --manifest-url=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452488    2568 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452492    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452502    2568 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452506    2568 flags.go:64] FLAG: --max-pods="110"
Nov 03 21:14:15.458217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452509    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452512    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452515    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452518    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452521    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452523    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452526    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452533    2568 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452537    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452540    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452543    2568 flags.go:64] FLAG: --pod-cidr=""
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452548    2568 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452550    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452557    2568 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452560    2568 flags.go:64] FLAG: --port="10250"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452563    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452566    2568 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-023f3183d384f3e57"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452569    2568 flags.go:64] FLAG: --qos-reserved=""
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452572    2568 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452575    2568 flags.go:64] FLAG: --register-node="true"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452578    2568 flags.go:64] FLAG: --register-with-taints=""
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452582    2568 flags.go:64] FLAG: --registry-burst="10"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452585    2568 flags.go:64] FLAG: --registry-qps="5"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452587    2568 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452590    2568 flags.go:64] FLAG: --reserved-memory=""
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452595    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 21:14:15.458835 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452598    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452601    2568 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452603    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452606    2568 flags.go:64] FLAG: --runonce="false"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452609    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452617    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452620    2568 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452623    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452626    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452629    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452632    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452635    2568 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452638    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452641    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452643    2568 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452646    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452649    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452653    2568 flags.go:64] FLAG: --system-cgroups=""
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452656    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452661    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452664    2568 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452667    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452673    2568 flags.go:64] FLAG: --tls-min-version=""
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452676    2568 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452679    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 21:14:15.459445 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452682    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452684    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452687    2568 flags.go:64] FLAG: --v="2"
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452692    2568 flags.go:64] FLAG: --version="false"
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452696    2568 flags.go:64] FLAG: --vmodule=""
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452700    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.452703    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452893    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452897    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452900    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452902    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452905    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452907    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452923    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452926    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452930    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452932    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452935    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452937    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452940    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452942    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.460218 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452945    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452947    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452950    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452954    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452956    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452959    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452961    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452964    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452966    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452969    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452971    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452974    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452976    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452979    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452981    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452985    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452989    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452992    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452994    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.460981 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.452998    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453000    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453002    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453005    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453007    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453010    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453018    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453022    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453025    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453027    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453029    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453032    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453034    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453037    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453040    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453042    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453046    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453049    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453051    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.461458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453053    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453056    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453059    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453061    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453064    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453066    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453069    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453071    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453074    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453076    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453078    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453081    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453083    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453086    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453089    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453091    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453093    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453096    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453098    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453101    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.461947 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453110    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453113    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453115    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453118    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453120    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453123    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453125    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453128    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453130    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453134    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453137    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453139    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453141    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453144    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453146    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453150    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453154    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453157    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453159    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.462480 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453162    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453164    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453167    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453170    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453172    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453175    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453177    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453179    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453182    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453184    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453187    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453189    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453192    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453196    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453204    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.453231    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:15.462932 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453279    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453283    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453286    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453288    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453291    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453293    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453296    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453299    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453301    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453304    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453306    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453309    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453312    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453314    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453316    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453319    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453321    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453323    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453339    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453342    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.463320 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453345    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453347    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453350    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453352    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453354    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453357    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453360    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453362    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453364    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453367    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453371    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453380    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453383    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453386    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453388    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453391    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453393    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453395    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453399    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.463809 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453403    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453406    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453409    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453412    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453415    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453419    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453421    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453424    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453426    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453429    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453431    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453433    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453436    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453438    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453441    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453444    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453446    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453449    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453451    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453454    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.464264 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453456    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453459    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453461    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453466    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453468    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453480    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453483    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453485    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453488    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453490    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453493    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453495    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453498    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453500    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453503    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453505    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453508    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453510    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453513    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.464736 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453515    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453518    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453520    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453523    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453525    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453528    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453530    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453532    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453536    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453539    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453542    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453545    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453547    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453550    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453552    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453555    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453559    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453561    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453564    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.465181 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453572    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453575    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453577    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453580    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453582    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453585    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453588    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453590    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.453593    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.461986    2568 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.462007    2568 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.462023    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.462028    2568 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462099    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462106    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462109    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462112    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462115    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462118    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.465641 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462121    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462123    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462126    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462129    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462131    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462134    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462136    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462139    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462141    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462145    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462148    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462150    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462153    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462155    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462158    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462169    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462172    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462175    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462178    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462181    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.466102 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462183    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462186    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462189    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462192    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462195    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462198    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462200    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462203    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462206    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462208    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462211    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462213    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462216    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462218    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462221    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462223    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462226    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462228    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462232    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.466596 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462236    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462240    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462243    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462246    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462249    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462252    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462254    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462257    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462259    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462262    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462264    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462267    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462269    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462272    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462274    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462277    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462280    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462283    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462286    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462289    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.467095 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462291    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462294    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462297    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462299    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462302    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462304    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462307    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462309    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462311    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462314    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462316    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462319    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462321    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462341    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462348    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462351    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462354    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462356    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462359    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.467682 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462362    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462364    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462366    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462369    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462371    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462374    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462376    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462379    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462381    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462385    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462387    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462390    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462393    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462395    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462398    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462401    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462404    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462406    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462409    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462412    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.468126 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462414    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462417    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.462449    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462599    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462604    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462607    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462609    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462612    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462615    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462618    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462620    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462623    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462625    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462628    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462630    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462632    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462635    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.468607 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462637    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462640    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462642    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462644    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462647    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462650    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462652    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462655    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462657    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462660    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462662    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462665    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462668    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462670    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462672    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462675    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462677    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462680    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462682    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.469041 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462685    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462687    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462689    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462692    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462695    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462699    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462703    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462707    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462711    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462714    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462717    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462721    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462724    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462727    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462730    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462733    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462736    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462739    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462742    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.469533 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462746    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462748    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462751    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462753    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462756    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462758    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462760    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462763    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462765    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462768    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462770    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462773    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462775    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462777    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462780    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462782    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462785    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462787    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462789    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.469984 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462793    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462796    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462798    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462801    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462803    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462806    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462808    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462811    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462813    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462815    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462818    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462821    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462824    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462826    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462829    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462832    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462834    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462837    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462839    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.470458 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462842    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462844    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462847    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462850    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462852    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462855    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462857    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462860    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462862    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462865    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462867    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462870    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462872    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462874    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462877    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462880    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.470900 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.462903    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462937    2568 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462941    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462945    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462947    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462950    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462953    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462955    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462961    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462963    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462966    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462969    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462971    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462974    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462977    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462979    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462982    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462984    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462987    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462989    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:14:15.471280 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462992    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462994    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.462997    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463000    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463002    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463005    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463007    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463010    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463012    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463015    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463017    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463020    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463023    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463028    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463030    2568 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463033    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463035    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463038    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463040    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463043    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463048    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:14:15.471871 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463051    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463053    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463056    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463059    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463061    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463064    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463066    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463069    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463071    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463074    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463076    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463078    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463081    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463083    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463086    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463088    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463092    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463096    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463099    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:14:15.472371 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463102    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463105    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463109    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463113    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463117    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463121    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463124    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463127    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463130    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463132    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463135    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463138    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463142    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463145    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463147    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463149    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463152    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463154    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463157    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:14:15.472808 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463159    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463161    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463164    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463167    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463169    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463172    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463174    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463176    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463179    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463181    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463184    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463186    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463189    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463191    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463194    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463197    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463199    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463202    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463207    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:14:15.473277 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463209    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463212    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463214    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463218    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463220    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463223    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463226    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463229    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:15.463231    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.463295    2568 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.465428    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.466373    2568 server.go:1014] "Starting client certificate rotation"
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.466471    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.467142    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 21:14:15.473743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.468769    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:14:15.474289 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.474274    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:14:15.484511 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.484491    2568 log.go:25] "Validated CRI v1 runtime API"
Nov 03 21:14:15.487267 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.487248    2568 log.go:25] "Validated CRI v1 image API"
Nov 03 21:14:15.488141 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.488117    2568 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 21:14:15.493217 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.493195    2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 919f8396-d4f3-4bca-a0d9-06d1eca19f80:/dev/nvme0n1p3 fd84755a-d72a-45e9-b697-9ca167a9f13c:/dev/nvme0n1p4]
Nov 03 21:14:15.493317 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.493217    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 21:14:15.499378 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.499219    2568 manager.go:217] Machine: {Timestamp:2026-11-03 21:14:15.497629656 +0000 UTC m=+0.347328653 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105817 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec221a41e85295bfb5c1a6bcdf991939 SystemUUID:ec221a41-e852-95bf-b5c1-a6bcdf991939 BootID:6bc352d1-18e1-47e5-a6ac-595058c0f924 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0d:67:d9:64:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0d:67:d9:64:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:21:f9:47:97:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 21:14:15.499943 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.499928    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 21:14:15.500073 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.500049    2568 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 21:14:15.504497 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.504467    2568 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 21:14:15.504631 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.504497    2568 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-65.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 21:14:15.504692 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.504634    2568 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 21:14:15.504692 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.504643    2568 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 21:14:15.504692 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.504684    2568 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:14:15.504769 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.504704    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:14:15.505631 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.505620    2568 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 21:14:15.509569 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.509555    2568 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 21:14:15.509701 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.509691    2568 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 21:14:15.510378 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.510368    2568 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 21:14:15.510409 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.510383    2568 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 21:14:15.510409 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.510394    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 21:14:15.510409 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.510403    2568 kubelet.go:405] "Adding apiserver pod source"
Nov 03 21:14:15.510519 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.510413    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 21:14:15.510763 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.510747    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:14:15.513103 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.513084    2568 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 21:14:15.517575 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.517559    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:14:15.518571 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518558    2568 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 21:14:15.518609 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518590    2568 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 21:14:15.518782 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518767    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 21:14:15.518811 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518795    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 21:14:15.518811 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518806    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 21:14:15.518811 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518812    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 21:14:15.518897 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518818    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 21:14:15.518897 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.518824    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 21:14:15.519220 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.519211    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 21:14:15.519254 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.519221    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 21:14:15.519254 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.519231    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 21:14:15.519254 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.519238    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 21:14:15.519254 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.519246    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 21:14:15.519254 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.519255    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 21:14:15.520157 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.520147    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 21:14:15.520194 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.520161    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 21:14:15.524067 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.524052    2568 server.go:1263] "Started kubelet"
Nov 03 21:14:15.524162 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.524092    2568 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 21:14:15.524258 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.524204    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 21:14:15.524311 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.524295    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 21:14:15.525077 ip-10-0-129-65 systemd[1]: Started Kubernetes Kubelet.
Nov 03 21:14:15.525916 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.525884    2568 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 21:14:15.527846 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.527831    2568 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 21:14:15.528893 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.528873    2568 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 21:14:15.530928 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.530909    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 21:14:15.531343 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.531312    2568 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 21:14:15.531549 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.531495    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:15.532097 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532081    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 21:14:15.532097 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532083    2568 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 21:14:15.532215 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532108    2568 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 21:14:15.532215 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532174    2568 factory.go:55] Registering systemd factory
Nov 03 21:14:15.532215 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532190    2568 factory.go:223] Registration of the systemd container factory successfully
Nov 03 21:14:15.532497 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532441    2568 factory.go:153] Registering CRI-O factory
Nov 03 21:14:15.532497 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532456    2568 factory.go:223] Registration of the crio container factory successfully
Nov 03 21:14:15.532600 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532508    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 21:14:15.532600 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532529    2568 factory.go:103] Registering Raw factory
Nov 03 21:14:15.532600 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532546    2568 manager.go:1199] Started watching for new ooms in manager
Nov 03 21:14:15.532739 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532715    2568 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 21:14:15.532739 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.532725    2568 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 21:14:15.533625 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.533608    2568 manager.go:319] Starting recovery of all containers
Nov 03 21:14:15.545021 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.544999    2568 manager.go:324] Recovery completed
Nov 03 21:14:15.557043 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.557027    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.559021 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559000    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.559093 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559037    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.559093 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559049    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.559633 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559618    2568 cpu_manager.go:226] "Starting" policy="none"
Nov 03 21:14:15.559633 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559630    2568 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 21:14:15.559633 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559619    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.559779 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.559651    2568 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 21:14:15.560474 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.560458    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.560530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.560489    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.560530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.560499    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.561685 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.561669    2568 policy_none.go:50] "Start"
Nov 03 21:14:15.561685 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.561687    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 21:14:15.561803 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.561696    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 21:14:15.563620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.563609    2568 policy_none.go:44] "Start"
Nov 03 21:14:15.597775 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.597737    2568 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 21:14:15.600227 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600214    2568 manager.go:341] "Starting Device Plugin manager"
Nov 03 21:14:15.600298 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.600254    2568 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 21:14:15.600298 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600268    2568 server.go:87] "Starting device plugin registration server"
Nov 03 21:14:15.600568 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600556    2568 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 21:14:15.600598 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600572    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 21:14:15.600703 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600684    2568 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 21:14:15.600830 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600815    2568 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 21:14:15.600830 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.600826    2568 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 21:14:15.601289 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.601226    2568 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 21:14:15.601289 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.601275    2568 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:15.668828 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.668790    2568 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-65.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 21:14:15.670751 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.668789    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-65.us-west-1.compute.internal.18e4a570cec1a108  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-65.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-65.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:15.524016392 +0000 UTC m=+0.373715389,LastTimestamp:2026-11-03 21:14:15.524016392 +0000 UTC m=+0.373715389,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-65.us-west-1.compute.internal,}"
Nov 03 21:14:15.673248 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.673221    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 21:14:15.674741 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.674723    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 21:14:15.674816 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.674746    2568 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 21:14:15.674816 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.674764    2568 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 21:14:15.674816 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.674800    2568 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 21:14:15.701704 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.701680    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.702677 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.702660    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.702781 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.702691    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.702781 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.702703    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.702781 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.702729    2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.717367 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.717342    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-65.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 21:14:15.775620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.775583    2568 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"]
Nov 03 21:14:15.775815 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.775662    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.777861 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.777794    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.777861 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.777830    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.777861 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.777848    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.780196 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.780177    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.780340 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.780307    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.780389 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.780362    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.781035 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.781020    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.781108 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.781053    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.781108 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.781064    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.781108 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.781084    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.781108 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.781096    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.781233 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.781069    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.783074 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.783058    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.783162 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.783088    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:15.783872 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.783855    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:15.783966 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.783884    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:15.783966 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.783898    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:15.785737 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.785718    2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 21:14:15.785829 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.785767    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x54ss"
Nov 03 21:14:15.806922 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.806900    2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.809134 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.809117    2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.834975 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.834907    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.835050 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.834980    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.835050 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.834998    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29b854c94d02e7fd862ea1754302117f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"29b854c94d02e7fd862ea1754302117f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.836058 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.836039    2568 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 21:14:15.881101 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.881034    2568 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 21:14:15.881101 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.881044    2568 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-65.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 21:14:15.881226 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.881066    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-65.us-west-1.compute.internal.18e4a570d0d7c6cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-65.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-65.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-65.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:15.559022283 +0000 UTC m=+0.408721279,LastTimestamp:2026-11-03 21:14:15.559022283 +0000 UTC m=+0.408721279,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-65.us-west-1.compute.internal,}"
Nov 03 21:14:15.888493 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.888473    2568 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-65.us-west-1.compute.internal\" not found" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.932915 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.932893    2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:15.936007 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.935987    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.936060 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.936016    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.936060 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.936034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29b854c94d02e7fd862ea1754302117f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"29b854c94d02e7fd862ea1754302117f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.936134 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.936077    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.936134 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.936096    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29b854c94d02e7fd862ea1754302117f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"29b854c94d02e7fd862ea1754302117f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.936207 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:15.936077    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:15.973898 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:15.973870    2568 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-65.us-west-1.compute.internal\" not found" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:16.088582 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.088542    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:16.089571 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.089553    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:16.089658 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.089583    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:16.089658 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.089593    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:16.089658 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.089621    2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:16.110781 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.110761    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:16.111850 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.111835    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:16.147416 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.147186    2568 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:16.147416 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.147387    2568 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-65.us-west-1.compute.internal\": node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.260093 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.260057    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.360671 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.360624    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.461673 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.461636    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.466807 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.466784    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 21:14:16.467008 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.466988    2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 21:14:16.467064 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.466976    2568 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a272bc74d03104c58bb06e21328e27dc-41a6df2b1383438f.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-65.us-west-1.compute.internal.18e4a570d0d83ea6\": read tcp 10.0.129.65:37382->3.130.232.210:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-65.us-west-1.compute.internal.18e4a570d0d83ea6  default   5094 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-65.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-65.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-65.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 21:14:15 +0000 UTC,LastTimestamp:2026-11-03 21:14:15.70270816 +0000 UTC m=+0.552407157,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-65.us-west-1.compute.internal,}"
Nov 03 21:14:16.562772 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.562741    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.663495 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.663462    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.735944 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.735868    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:14:16.736712 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.736692    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:14:16.736786 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.736733    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:14:16.736786 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.736748    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:14:16.764481 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.764451    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.790140 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.790106    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:09:16 +0000 UTC" deadline="2028-05-19 12:28:48.016606198 +0000 UTC"
Nov 03 21:14:16.790140 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.790137    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13503h14m31.226473159s"
Nov 03 21:14:16.865197 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.865160    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.965592 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:16.965550    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:16.971517 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:16.971478    2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:17.066601 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.066555    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:17.091254 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.091227    2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:17.167121 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.167091    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:14:17.175978 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.175959    2568 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:17.218859 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.218813    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f624a3552b5bcbd4b3914b37486b5a.slice/crio-7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb WatchSource:0}: Error finding container 7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb: Status 404 returned error can't find the container with id 7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb
Nov 03 21:14:17.219066 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.219049    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b854c94d02e7fd862ea1754302117f.slice/crio-1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2 WatchSource:0}: Error finding container 1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2: Status 404 returned error can't find the container with id 1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2
Nov 03 21:14:17.227241 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.227223    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:14:17.232164 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.232146    2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:17.296215 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.296132    2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:17.297326 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.297309    2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:14:17.356856 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.356825    2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:14:17.474490 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.474458    2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:14:17.513183 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.513147    2568 apiserver.go:52] "Watching apiserver"
Nov 03 21:14:17.532712 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.532671    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 21:14:17.623653 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.623559    2568 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 21:14:17.623974 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.623950    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-fp2r2","openshift-multus/multus-gg868","openshift-network-diagnostics/network-check-target-kgjns","openshift-network-operator/iptables-alerter-wrgf2","openshift-image-registry/node-ca-pshfv","openshift-multus/network-metrics-daemon-tjq5q","kube-system/global-pull-secret-syncer-s77gw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm","openshift-cluster-node-tuning-operator/tuned-zv8ps","openshift-ovn-kubernetes/ovnkube-node-gmxs8","kube-system/konnectivity-agent-h4zdr","openshift-dns/node-resolver-g7cv4"]
Nov 03 21:14:17.627933 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.627908    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.628030 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.627993    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:17.630127 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.630108    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.632417 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.632397    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:17.632516 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.632455    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:17.634713 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.634698    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.637109 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.637091    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.639375 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.639360    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:17.639454 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.639435    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:17.642230 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.642211    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.644830 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.644811    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.644929 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.644844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb32df-a293-4351-8743-a3b12b67dd87-konnectivity-ca\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.644929 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.644871    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-host-slash\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.644929 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.644926    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.645100 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.644933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eeb32df-a293-4351-8743-a3b12b67dd87-agent-certs\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.645100 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.645012    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:17.645661 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.645262    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-iptables-alerter-script\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.645661 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.645650    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8wf\" (UniqueName: \"kubernetes.io/projected/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-kube-api-access-9b8wf\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.645919 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.645900    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-kubelet-config\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.646031 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.645978    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-dbus\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.647942 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.647920    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.650709 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.650600    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.653035 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.653011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.655545 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.655530    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.679662 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.679609    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"f2f624a3552b5bcbd4b3914b37486b5a","Type":"ContainerStarted","Data":"7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb"}
Nov 03 21:14:17.680981 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.680958    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"29b854c94d02e7fd862ea1754302117f","Type":"ContainerStarted","Data":"1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2"}
Nov 03 21:14:17.686149 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.686115    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lzq4j\""
Nov 03 21:14:17.686252 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.686184    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 21:14:17.686252 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.686243    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 21:14:17.690121 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.690103    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.690315 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.690300    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.690355 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.690327    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trdz7\""
Nov 03 21:14:17.690495 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.690482    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 21:14:17.692422 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.692386    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.692507 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.692449    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 21:14:17.692507 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.692478    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4rzx\""
Nov 03 21:14:17.692587 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.692510    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.697887 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.697865    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.697961 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.697864    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.697961 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.697912    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cldzt\""
Nov 03 21:14:17.700652 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.700631    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.700759 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.700733    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.700996 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.700981    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxfhg\""
Nov 03 21:14:17.701128 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.701113    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 21:14:17.701164 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.701120    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 21:14:17.701164 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.701146    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 21:14:17.702991 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.702976    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 21:14:17.733911 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.733879    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzfft\""
Nov 03 21:14:17.734259 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.734242    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 21:14:17.746343 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746295    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovn-node-metrics-cert\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.746343 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746327    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d1e36e3-40f2-4c43-a009-e323301c57e6-host\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.746453 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746362    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.746453 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746406    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56dd4c25-1333-40f6-867d-1f02ff44b0d9-hosts-file\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.746453 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746427    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mtm\" (UniqueName: \"kubernetes.io/projected/d3648276-55d9-46b7-ac52-ea7223dd5868-kube-api-access-d4mtm\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.746453 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746450    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-multus-certs\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.746649 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746488    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-netns\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.746649 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746548    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-var-lib-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.746649 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746599    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8wf\" (UniqueName: \"kubernetes.io/projected/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-kube-api-access-9b8wf\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.746794 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746709    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-lib-modules\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.746794 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746743    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-host\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.746794 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-script-lib\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.746794 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746778    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.746970 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746805    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-systemd\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.746970 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746842    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-sys\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.746970 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746874    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-log-socket\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.746970 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746927    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkmh\" (UniqueName: \"kubernetes.io/projected/aa443bcf-d427-46f2-b924-beb20bd4df7b-kube-api-access-5xkmh\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.746970 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746948    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-system-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.746970 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.746968    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-kubelet\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.747266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747011    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-node-log\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.747266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747055    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.747266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747096    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.747266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747116    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-cnibin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.747266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747160    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn8n\" (UniqueName: \"kubernetes.io/projected/eb673cf5-6137-4a72-8910-e957e3685a71-kube-api-access-tjn8n\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.747266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747206    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-systemd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.747538 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-iptables-alerter-script\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.747538 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747306    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-tmp\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.747538 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747428    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-multus\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.747538 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747460    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-multus-daemon-config\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.747538 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747506    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-systemd-units\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.747772 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-etc-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.747772 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747598    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:17.747772 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747641    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-run\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.747772 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747672    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-k8s-cni-cncf-io\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.747772 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747701    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-kubelet\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.747772 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.747945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dd4c25-1333-40f6-867d-1f02ff44b0d9-tmp-dir\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.747945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747869    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-socket-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.747945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747901    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-registration-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.747945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747920    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-iptables-alerter-script\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.748074 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747951    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.748074 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747978    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d1e36e3-40f2-4c43-a009-e323301c57e6-serviceca\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.748074 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.747994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.748074 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748025    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb32df-a293-4351-8743-a3b12b67dd87-konnectivity-ca\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.748193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748081    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysconfig\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.748193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748143    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-conf-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.748193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:17.748193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748185    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.748310 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748209    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-sys-fs\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.748310 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-cni-binary-copy\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.748310 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748286    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-socket-dir-parent\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.748436 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748313    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-ovn\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.748436 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748411    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-kubelet-config\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.748499 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-dbus\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.748532 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748504    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-host-slash\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.748532 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748513    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-kubelet-config\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.748593 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748534    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-conf\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.748593 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748563    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-bin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.748654 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748612    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb32df-a293-4351-8743-a3b12b67dd87-konnectivity-ca\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.748654 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748619    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-etc-kubernetes\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.748720 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-bin\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.748720 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748680    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-host-slash\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.748720 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748700    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-dbus\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.748811 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748723    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-netd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.748811 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748760    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkzbq\" (UniqueName: \"kubernetes.io/projected/56dd4c25-1333-40f6-867d-1f02ff44b0d9-kube-api-access-pkzbq\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.748811 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748794    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.748899 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748826    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.748899 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-var-lib-kubelet\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.748899 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-slash\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.748989 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748896    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.748989 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748915    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-config\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.748989 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748952    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:17.748989 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748976    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-tuned\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.749105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.748993    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdl9\" (UniqueName: \"kubernetes.io/projected/0d1e36e3-40f2-4c43-a009-e323301c57e6-kube-api-access-ptdl9\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.749105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749028    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkhz\" (UniqueName: \"kubernetes.io/projected/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-kube-api-access-5vkhz\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.749105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749087    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eeb32df-a293-4351-8743-a3b12b67dd87-agent-certs\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.749193 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.749116    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:17.749193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrdx\" (UniqueName: \"kubernetes.io/projected/be40fcfd-b776-4ede-ad34-e4be87a007e0-kube-api-access-lkrdx\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:17.749193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749153    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksds\" (UniqueName: \"kubernetes.io/projected/5718e9a0-0924-41fe-a482-557054019ca9-kube-api-access-lksds\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.749327 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.749193    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:14:18.249153274 +0000 UTC m=+3.098852278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:17.749327 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749223    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-modprobe-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.749327 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-hostroot\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.749945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-env-overrides\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.749945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749684    2568 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 21:14:17.749945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749723    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-cnibin\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.749945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749808    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.749945 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749854    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-device-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.750959 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749941    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-kubernetes\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.750959 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.749998    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-netns\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.755125 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.755097    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eeb32df-a293-4351-8743-a3b12b67dd87-agent-certs\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.787475 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.787451    2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 21:14:17.787606 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.787530    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d4fbc"
Nov 03 21:14:17.794755 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.794726    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 21:14:17.794850 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.794775    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 21:14:17.795061 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795029    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4f9ng\""
Nov 03 21:14:17.795606 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795590    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 21:14:17.795682 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795614    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.795682 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795632    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.795682 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795664    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.795682 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795671    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 21:14:17.795682 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795672    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbwk2\""
Nov 03 21:14:17.795879 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795697    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.795879 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795722    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 21:14:17.795879 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795735    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbt6z\""
Nov 03 21:14:17.795879 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795741    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 21:14:17.795879 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.795775    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 21:14:17.805525 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.805502    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:17.805525 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.805523    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:17.805639 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.805534    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:17.805639 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.805600    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:18.30558525 +0000 UTC m=+3.155284251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:17.807171 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.807150    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8wf\" (UniqueName: \"kubernetes.io/projected/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-kube-api-access-9b8wf\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.850878 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850830    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.850878 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850881    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-systemd\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850900    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-sys\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-log-socket\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850954    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkmh\" (UniqueName: \"kubernetes.io/projected/aa443bcf-d427-46f2-b924-beb20bd4df7b-kube-api-access-5xkmh\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850973    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-system-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850971    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-systemd\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.850972    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-sys\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.851058 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851016    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-log-socket\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851077    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-system-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-kubelet\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851151    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-node-log\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851179    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851182    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-kubelet\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851206    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851226    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-node-log\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-cnibin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn8n\" (UniqueName: \"kubernetes.io/projected/eb673cf5-6137-4a72-8910-e957e3685a71-kube-api-access-tjn8n\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851289    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-systemd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851314    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-tmp\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.851336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851340    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-multus\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851373    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851390    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-multus-daemon-config\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851398    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-cnibin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851419    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-systemd-units\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-etc-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851451    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-systemd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851475    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851493    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-multus\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-run\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-k8s-cni-cncf-io\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851557    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-kubelet\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851581    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851590    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851609    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dd4c25-1333-40f6-867d-1f02ff44b0d9-tmp-dir\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851636    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-socket-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.851695    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:17.851909 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851744    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-kubelet\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:17.851764    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:18.351744627 +0000 UTC m=+3.201443616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851763    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-socket-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851788    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-systemd-units\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-registration-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851748    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851851    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-k8s-cni-cncf-io\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851852    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-registration-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851855    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851854    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851905    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-run\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851952    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-etc-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.851999    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d1e36e3-40f2-4c43-a009-e323301c57e6-serviceca\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852017    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dd4c25-1333-40f6-867d-1f02ff44b0d9-tmp-dir\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysconfig\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852081    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-conf-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.852785 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852104    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852109    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852125    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852170    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysconfig\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852169    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-conf-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852186    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852208    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-sys-fs\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852240    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-cni-binary-copy\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852265    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-socket-dir-parent\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852291    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-ovn\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852316    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-sys-fs\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-conf\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852338    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-socket-dir-parent\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852346    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-bin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852290    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-multus-daemon-config\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-etc-kubernetes\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852396    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-ovn\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852424    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-bin\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.853597 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852433    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-bin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852444    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-etc-kubernetes\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852450    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-netd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852478    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkzbq\" (UniqueName: \"kubernetes.io/projected/56dd4c25-1333-40f6-867d-1f02ff44b0d9-kube-api-access-pkzbq\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852492    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-bin\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852505    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852532    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852537    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-netd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852557    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-var-lib-kubelet\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852567    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d1e36e3-40f2-4c43-a009-e323301c57e6-serviceca\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852581    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852584    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-conf\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852583    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-slash\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852621    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-slash\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852643    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852687    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-config\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852648    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-var-lib-kubelet\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.854620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852733    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-tuned\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852764    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdl9\" (UniqueName: \"kubernetes.io/projected/0d1e36e3-40f2-4c43-a009-e323301c57e6-kube-api-access-ptdl9\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkhz\" (UniqueName: \"kubernetes.io/projected/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-kube-api-access-5vkhz\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852828    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrdx\" (UniqueName: \"kubernetes.io/projected/be40fcfd-b776-4ede-ad34-e4be87a007e0-kube-api-access-lkrdx\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852858    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksds\" (UniqueName: \"kubernetes.io/projected/5718e9a0-0924-41fe-a482-557054019ca9-kube-api-access-lksds\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852795    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-cni-binary-copy\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-modprobe-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-hostroot\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-env-overrides\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852980    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-cnibin\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853035    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-device-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.852704    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853060    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-kubernetes\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853081    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-modprobe-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853090    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-netns\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853139    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-netns\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.855530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853142    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovn-node-metrics-cert\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853180    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d1e36e3-40f2-4c43-a009-e323301c57e6-host\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853195    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-hostroot\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853210    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853240    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56dd4c25-1333-40f6-867d-1f02ff44b0d9-hosts-file\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853255    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-device-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853270    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mtm\" (UniqueName: \"kubernetes.io/projected/d3648276-55d9-46b7-ac52-ea7223dd5868-kube-api-access-d4mtm\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853309    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-multus-certs\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853338    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-netns\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-var-lib-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853421    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-lib-modules\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-host\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853475    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-script-lib\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853475    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-cnibin\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853552    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-netns\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853563    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d1e36e3-40f2-4c43-a009-e323301c57e6-host\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853570    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-kubernetes\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853600    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853675    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-lib-modules\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853732    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-multus-certs\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853738    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-host\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853757    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-var-lib-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853789    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56dd4c25-1333-40f6-867d-1f02ff44b0d9-hosts-file\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853812    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.853951    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-config\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.854012    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.854267    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-env-overrides\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.854344    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-script-lib\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.854453    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-tmp\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.855129    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-tuned\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.856874 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.856452    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovn-node-metrics-cert\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.913757 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.913667    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkmh\" (UniqueName: \"kubernetes.io/projected/aa443bcf-d427-46f2-b924-beb20bd4df7b-kube-api-access-5xkmh\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.913935 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.913800    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn8n\" (UniqueName: \"kubernetes.io/projected/eb673cf5-6137-4a72-8910-e957e3685a71-kube-api-access-tjn8n\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.914525 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.914490    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkhz\" (UniqueName: \"kubernetes.io/projected/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-kube-api-access-5vkhz\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:17.914938 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.914911    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrdx\" (UniqueName: \"kubernetes.io/projected/be40fcfd-b776-4ede-ad34-e4be87a007e0-kube-api-access-lkrdx\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:17.915207 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.915186    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdl9\" (UniqueName: \"kubernetes.io/projected/0d1e36e3-40f2-4c43-a009-e323301c57e6-kube-api-access-ptdl9\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.915412 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.915382    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mtm\" (UniqueName: \"kubernetes.io/projected/d3648276-55d9-46b7-ac52-ea7223dd5868-kube-api-access-d4mtm\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.915716 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.915697    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksds\" (UniqueName: \"kubernetes.io/projected/5718e9a0-0924-41fe-a482-557054019ca9-kube-api-access-lksds\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.915750 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.915725    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkzbq\" (UniqueName: \"kubernetes.io/projected/56dd4c25-1333-40f6-867d-1f02ff44b0d9-kube-api-access-pkzbq\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.941925 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.941886    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:17.951493 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.951472    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:14:17.953005 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.952979    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeb32df_a293_4351_8743_a3b12b67dd87.slice/crio-0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3 WatchSource:0}: Error finding container 0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3: Status 404 returned error can't find the container with id 0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3
Nov 03 21:14:17.958043 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.958016    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:14:17.960510 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.960479    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bfe6b0_1d4f_47db_a462_a1a676845dcf.slice/crio-8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c WatchSource:0}: Error finding container 8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c: Status 404 returned error can't find the container with id 8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c
Nov 03 21:14:17.962120 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.962098    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:14:17.968394 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.968365    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1e36e3_40f2_4c43_a009_e323301c57e6.slice/crio-f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890 WatchSource:0}: Error finding container f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890: Status 404 returned error can't find the container with id f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890
Nov 03 21:14:17.971914 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.971887    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dd4c25_1333_40f6_867d_1f02ff44b0d9.slice/crio-fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec WatchSource:0}: Error finding container fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec: Status 404 returned error can't find the container with id fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec
Nov 03 21:14:17.979278 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.979248    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gg868"
Nov 03 21:14:17.985460 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.985432    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:14:17.989079 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.989057    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb673cf5_6137_4a72_8910_e957e3685a71.slice/crio-90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec WatchSource:0}: Error finding container 90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec: Status 404 returned error can't find the container with id 90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec
Nov 03 21:14:17.990054 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.990032    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:14:17.992240 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:17.992224    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:14:17.995256 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.995227    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5718e9a0_0924_41fe_a482_557054019ca9.slice/crio-9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8 WatchSource:0}: Error finding container 9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8: Status 404 returned error can't find the container with id 9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8
Nov 03 21:14:17.999865 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:17.999844    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa443bcf_d427_46f2_b924_beb20bd4df7b.slice/crio-cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b WatchSource:0}: Error finding container cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b: Status 404 returned error can't find the container with id cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b
Nov 03 21:14:18.003022 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:18.002992    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3648276_55d9_46b7_ac52_ea7223dd5868.slice/crio-ee13442af2119120618a5aac0103dfd5b0ba094a9419cc4142f0ef6750653146 WatchSource:0}: Error finding container ee13442af2119120618a5aac0103dfd5b0ba094a9419cc4142f0ef6750653146: Status 404 returned error can't find the container with id ee13442af2119120618a5aac0103dfd5b0ba094a9419cc4142f0ef6750653146
Nov 03 21:14:18.010499 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.010475    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:18.020269 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:14:18.020246    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae80c711_2bb7_43e4_9bf2_521dbc9e4291.slice/crio-05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831 WatchSource:0}: Error finding container 05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831: Status 404 returned error can't find the container with id 05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831
Nov 03 21:14:18.256350 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.256292    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:18.256548 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.256482    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:18.256602 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.256563    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:14:19.256541677 +0000 UTC m=+4.106240668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:18.364009 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.363974    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:18.364250 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.364230    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:18.364503 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.364485    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:18.364837 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.364820    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:18.364902 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.364845    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:18.364902 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.364860    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:18.365131 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.365111    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:19.36490032 +0000 UTC m=+4.214599322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:18.365182 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.365146    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:19.365133269 +0000 UTC m=+4.214832260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:18.675529 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.675435    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:18.676247 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:18.675583    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:18.688253 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.686238    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g7cv4" event={"ID":"56dd4c25-1333-40f6-867d-1f02ff44b0d9","Type":"ContainerStarted","Data":"fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec"}
Nov 03 21:14:18.688253 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.687748    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wrgf2" event={"ID":"d8bfe6b0-1d4f-47db-a462-a1a676845dcf","Type":"ContainerStarted","Data":"8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c"}
Nov 03 21:14:18.688901 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.688843    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4zdr" event={"ID":"5eeb32df-a293-4351-8743-a3b12b67dd87","Type":"ContainerStarted","Data":"0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3"}
Nov 03 21:14:18.690089 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.690001    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831"}
Nov 03 21:14:18.692000 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.691969    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zv8ps" event={"ID":"d3648276-55d9-46b7-ac52-ea7223dd5868","Type":"ContainerStarted","Data":"ee13442af2119120618a5aac0103dfd5b0ba094a9419cc4142f0ef6750653146"}
Nov 03 21:14:18.693341 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.693317    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerStarted","Data":"9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8"}
Nov 03 21:14:18.694664 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.694638    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gg868" event={"ID":"eb673cf5-6137-4a72-8910-e957e3685a71","Type":"ContainerStarted","Data":"90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec"}
Nov 03 21:14:18.695865 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.695842    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pshfv" event={"ID":"0d1e36e3-40f2-4c43-a009-e323301c57e6","Type":"ContainerStarted","Data":"f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890"}
Nov 03 21:14:18.697138 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.697114    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm" event={"ID":"aa443bcf-d427-46f2-b924-beb20bd4df7b","Type":"ContainerStarted","Data":"cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b"}
Nov 03 21:14:18.791747 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.791687    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:18 +0000 UTC" deadline="2028-07-17 09:55:55.320134174 +0000 UTC"
Nov 03 21:14:18.791747 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:18.791732    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14916h41m36.528405658s"
Nov 03 21:14:19.273921 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.273874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:19.274123 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.274042    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:19.274190 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.274123    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:14:21.274102278 +0000 UTC m=+6.123801268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.375001    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.375058    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.375219    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.375241    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.375256    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.375318    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:19.375360 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.375331    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:21.375310173 +0000 UTC m=+6.225009177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:19.375971 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.375432    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:21.375391993 +0000 UTC m=+6.225090981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:19.676172 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.676091    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:19.676846 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.676279    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:19.676846 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.676530    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:19.676846 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:19.676640    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:19.792891 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.792842    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:18 +0000 UTC" deadline="2028-05-20 12:58:00.014894266 +0000 UTC"
Nov 03 21:14:19.792891 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:19.792888    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13527h43m40.222012146s"
Nov 03 21:14:20.676291 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:20.676052    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:20.676291 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:20.676226    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:21.291963 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.291717    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:21.292148 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.292083    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:21.292214 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.292168    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:14:25.292147343 +0000 UTC m=+10.141846355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.392741    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.392806    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.392975    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.392996    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.393009    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.393074    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:25.393054411 +0000 UTC m=+10.242753413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.393156    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:21.393282 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.393198    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:25.393184678 +0000 UTC m=+10.242883683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:21.687191 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.687156    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:21.687702 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.687296    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:21.687702 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.687391    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:21.687702 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:21.687493    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:21.744938 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.744883    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"29b854c94d02e7fd862ea1754302117f","Type":"ContainerStarted","Data":"cf66ecd86aa43b38ec7e23e6745d4a78640b6196646f72280845862537d00cbb"}
Nov 03 21:14:21.752609 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.752452    2568 generic.go:356] "Generic (PLEG): container finished" podID="f2f624a3552b5bcbd4b3914b37486b5a" containerID="476dcf37c26782319d2e02475ca3c536352ac5b86b9f5f3b7bcba4c42376fd20" exitCode=0
Nov 03 21:14:21.752609 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.752505    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"f2f624a3552b5bcbd4b3914b37486b5a","Type":"ContainerDied","Data":"476dcf37c26782319d2e02475ca3c536352ac5b86b9f5f3b7bcba4c42376fd20"}
Nov 03 21:14:21.992010 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:21.991634    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal" podStartSLOduration=3.991616594 podStartE2EDuration="3.991616594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:21.868304289 +0000 UTC m=+6.718003300" watchObservedRunningTime="2026-11-03 21:14:21.991616594 +0000 UTC m=+6.841315602"
Nov 03 21:14:22.675746 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:22.675711    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:22.675942 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:22.675863    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:22.763589 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:22.763543    2568 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="d6027ce2220e8a691d54b0374bc94b2a3569bddfaa029b5a63862e99d6ad43be" exitCode=0
Nov 03 21:14:22.764069 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:22.763675    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerDied","Data":"d6027ce2220e8a691d54b0374bc94b2a3569bddfaa029b5a63862e99d6ad43be"}
Nov 03 21:14:22.772985 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:22.772952    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"f2f624a3552b5bcbd4b3914b37486b5a","Type":"ContainerStarted","Data":"7ea07e6f822a5b939db961b17be5d6f8e3cdd8c7cd0b08ee2506c9be0c92de99"}
Nov 03 21:14:23.068500 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:23.068335    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" podStartSLOduration=5.068315312 podStartE2EDuration="5.068315312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:23.067543318 +0000 UTC m=+7.917242328" watchObservedRunningTime="2026-11-03 21:14:23.068315312 +0000 UTC m=+7.918014322"
Nov 03 21:14:23.678641 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:23.678605    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:23.678793 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:23.678717    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:23.679187 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:23.678891    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:23.682790 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:23.682750    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:23.778328 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:23.778265    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4zdr" event={"ID":"5eeb32df-a293-4351-8743-a3b12b67dd87","Type":"ContainerStarted","Data":"8fcafb7df5086d237234d863211cbd9f996257147cbe5b139e7e100c6240e4d9"}
Nov 03 21:14:24.675668 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:24.675621    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:24.675863 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:24.675776    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:24.785292 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:24.785254    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pshfv" event={"ID":"0d1e36e3-40f2-4c43-a009-e323301c57e6","Type":"ContainerStarted","Data":"a42078431ad52e92cf58e7fcdf13cb9038605691b11c9e7f1fb994dddb813db4"}
Nov 03 21:14:24.908981 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:24.908683    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h4zdr" podStartSLOduration=2.869976195 podStartE2EDuration="7.908661841s" totalImagesPullingTime="5.038685646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:23.899578189 +0000 UTC m=+8.749277198" watchObservedRunningTime="2026-11-03 21:14:24.908661841 +0000 UTC m=+9.758360855"
Nov 03 21:14:25.328113 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:25.327779    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:25.328113 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.327923    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:25.328113 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.327985    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:14:33.327968017 +0000 UTC m=+18.177667008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:25.428402    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:25.428465    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.428630    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.428651    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.428664    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.428727    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:33.428707033 +0000 UTC m=+18.278406025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.428802    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:25.428896 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.428840    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:33.428827737 +0000 UTC m=+18.278526728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:25.679103 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:25.679003    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:25.679251 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.679118    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:25.679251 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:25.679186    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:25.679328 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:25.679283    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:25.791530 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:25.791486    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm" event={"ID":"aa443bcf-d427-46f2-b924-beb20bd4df7b","Type":"ContainerStarted","Data":"3abb920782e945ccd6c8807dd0a42dcc43cfb1f3f143cb334981179ceabe6c95"}
Nov 03 21:14:26.341375 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:26.341330    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:26.342605 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:26.342577    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:26.468349 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:26.468274    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pshfv" podStartSLOduration=3.762294615 podStartE2EDuration="9.468252122s" totalImagesPullingTime="5.705957507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:24.908579276 +0000 UTC m=+9.758278290" watchObservedRunningTime="2026-11-03 21:14:26.468252122 +0000 UTC m=+11.317951132"
Nov 03 21:14:26.676185 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:26.676093    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:26.676378 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:26.676253    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:27.226209 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.226151    2568 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 21:14:27.611586 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.610816    2568 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:14:27.226176871Z","UUID":"ef4553be-68c7-4641-9154-8f086cfa0cfb","Handler":null,"Name":"","Endpoint":""}
Nov 03 21:14:27.612705 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.612684    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 21:14:27.612848 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.612715    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 21:14:27.675277 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.675245    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:27.675431 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:27.675390    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:27.675526 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.675431    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:27.675526 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:27.675508    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:27.798978 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.798942    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerStarted","Data":"2cc26764ce0b508e27e20775151cbb1af7e771a1f9c72da6f0989dda8fac3ca7"}
Nov 03 21:14:27.802000 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.801961    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gg868" event={"ID":"eb673cf5-6137-4a72-8910-e957e3685a71","Type":"ContainerStarted","Data":"7c542efbd051e681b8d3b84f8859ee25158a7e658dfa8dcb4606530abc5f4ce2"}
Nov 03 21:14:27.810635 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.810593    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm" event={"ID":"aa443bcf-d427-46f2-b924-beb20bd4df7b","Type":"ContainerStarted","Data":"d8d4d1e5e74d94175e69d1b3437d2f8b65647ffef733e336abd9427fc1eb34d3"}
Nov 03 21:14:27.812483 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:27.812446    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g7cv4" event={"ID":"56dd4c25-1333-40f6-867d-1f02ff44b0d9","Type":"ContainerStarted","Data":"50ee88d8360e6f79dcc9e79251dc18c23a8a97c00ec25936489a1138475f3530"}
Nov 03 21:14:28.166376 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:28.166183    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g7cv4" podStartSLOduration=2.238755955 podStartE2EDuration="11.166165892s" totalImagesPullingTime="8.927409937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:28.04450126 +0000 UTC m=+12.894200269" watchObservedRunningTime="2026-11-03 21:14:28.166165892 +0000 UTC m=+13.015864901"
Nov 03 21:14:28.675719 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:28.675676    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:28.676156 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:28.675844    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:28.818742 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:28.818704    2568 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="2cc26764ce0b508e27e20775151cbb1af7e771a1f9c72da6f0989dda8fac3ca7" exitCode=0
Nov 03 21:14:28.819448 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:28.819416    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerDied","Data":"2cc26764ce0b508e27e20775151cbb1af7e771a1f9c72da6f0989dda8fac3ca7"}
Nov 03 21:14:28.945388 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:28.945323    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gg868" podStartSLOduration=3.221342025 podStartE2EDuration="11.94530491s" totalImagesPullingTime="8.723962885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:28.167014222 +0000 UTC m=+13.016713232" watchObservedRunningTime="2026-11-03 21:14:28.94530491 +0000 UTC m=+13.795003915"
Nov 03 21:14:29.676135 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:29.675944    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:29.676551 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:29.675979    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:29.676551 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:29.676236    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:29.676551 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:29.676303    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:29.823172 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:29.823142    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm" event={"ID":"aa443bcf-d427-46f2-b924-beb20bd4df7b","Type":"ContainerStarted","Data":"fe2779baf366c481e73604cfeb36a72c4e30aacef0389d06cf3a437ec160218e"}
Nov 03 21:14:29.824863 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:29.824839    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wrgf2" event={"ID":"d8bfe6b0-1d4f-47db-a462-a1a676845dcf","Type":"ContainerStarted","Data":"8236d86fc74caa11a701b07b4e300678d0e23998fb7eca1b059e4471e9444339"}
Nov 03 21:14:29.936564 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:29.936506    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm" podStartSLOduration=2.231220666 podStartE2EDuration="12.936487227s" totalImagesPullingTime="10.705266561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:29.936107891 +0000 UTC m=+14.785806897" watchObservedRunningTime="2026-11-03 21:14:29.936487227 +0000 UTC m=+14.786186236"
Nov 03 21:14:30.047872 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:30.047824    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wrgf2" podStartSLOduration=3.537277359 podStartE2EDuration="13.047809427s" totalImagesPullingTime="9.510532068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:30.047536998 +0000 UTC m=+14.897236003" watchObservedRunningTime="2026-11-03 21:14:30.047809427 +0000 UTC m=+14.897508434"
Nov 03 21:14:30.675504 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:30.675420    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:30.675669 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:30.675545    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:30.828692 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:30.828650    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zv8ps" event={"ID":"d3648276-55d9-46b7-ac52-ea7223dd5868","Type":"ContainerStarted","Data":"697ccc6120615fdee7d56765ae5b48b87a64c370e46744b6ee599c2c4373a694"}
Nov 03 21:14:30.832762 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:30.832733    2568 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="3deed723913e84bb2192875fae0e72c083e6a7ed552e964a3783d438fab7a384" exitCode=0
Nov 03 21:14:30.832907 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:30.832819    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerDied","Data":"3deed723913e84bb2192875fae0e72c083e6a7ed552e964a3783d438fab7a384"}
Nov 03 21:14:31.057195 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:31.057123    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zv8ps" podStartSLOduration=2.292671179 podStartE2EDuration="14.057103009s" totalImagesPullingTime="11.76443183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:30.941643758 +0000 UTC m=+15.791342763" watchObservedRunningTime="2026-11-03 21:14:31.057103009 +0000 UTC m=+15.906802014"
Nov 03 21:14:31.675836 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:31.675794    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:31.676026 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:31.675840    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:31.676026 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:31.675933    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:31.676138 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:31.676072    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:32.676099 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:32.676024    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:32.676709 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:32.676179    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:32.840123 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:32.840077    2568 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="e8030149cbce3636ac3b9d8706ee048032c7d0eff9302bde3158c5e86eda6357" exitCode=0
Nov 03 21:14:32.840292 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:32.840138    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerDied","Data":"e8030149cbce3636ac3b9d8706ee048032c7d0eff9302bde3158c5e86eda6357"}
Nov 03 21:14:33.395005 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:33.394963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:33.395189 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.395113    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:33.395189 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.395175    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.395160142 +0000 UTC m=+34.244859140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:33.496199 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:33.496155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:33.496199 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:33.496203    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:33.496449 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.496307    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:33.496449 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.496318    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:33.496449 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.496360    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:33.496449 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.496373    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:33.496449 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.496389    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.496374463 +0000 UTC m=+34.346073451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:33.496449 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.496419    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:14:49.496407087 +0000 UTC m=+34.346106075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:33.675726 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:33.675639    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:33.675726 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:33.675672    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:33.675969 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.675762    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:33.675969 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:33.675878    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:34.676011 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.675773    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:34.676503 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:34.676106    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:34.848228 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.848000    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"d59e333d4acf1725f331030f9b6bfacb857a4b977d744d613c163d051af60be0"}
Nov 03 21:14:34.848407 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.848295    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"ab7af3cf5bace8aae5127a03bab86f30a81f3d8d71d9d30b8118bcb0b51eaa22"}
Nov 03 21:14:34.848407 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.848364    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"4abf2925a9c6ebb9d8f2118a494c2dcfec802f61f32419db72a1d940248bc5dd"}
Nov 03 21:14:34.848563 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.848405    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"9a76bddb5613aa8ca71a05ea996b4d261200ac4d50e45d5ccec28cb9bd033c1b"}
Nov 03 21:14:34.848563 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.848450    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"f476a50fe90e5e0b770091e1bfb974fd601efe243bf92047aedfb19c4f8bedc2"}
Nov 03 21:14:34.848563 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.848498    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"f6824d8af94e6b4ccf71d133b8560ff21ecd10adeb22bb98058dc97998e6b3f0"}
Nov 03 21:14:34.850917 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:34.850891    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerStarted","Data":"cb5ea2d703cb0beccb4b3c08ebec5e745ab0d855811c0ac41838304156bf511f"}
Nov 03 21:14:35.676230 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:35.676195    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:35.676743 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:35.676350    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:35.676743 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:35.676384    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:35.676743 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:35.676485    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:35.857586 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:35.857554    2568 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="cb5ea2d703cb0beccb4b3c08ebec5e745ab0d855811c0ac41838304156bf511f" exitCode=0
Nov 03 21:14:35.857748 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:35.857602    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerDied","Data":"cb5ea2d703cb0beccb4b3c08ebec5e745ab0d855811c0ac41838304156bf511f"}
Nov 03 21:14:36.675595 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:36.675560    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:36.675756 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:36.675692    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:36.862979 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:36.862890    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"39ffa468e65bed717d342805837058890a25cad758b77f7f0db75bb7ae2113a7"}
Nov 03 21:14:36.865966 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:36.865936    2568 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="a34f00d7a6763d6be63467c7f34595921056010afa4a8cba99071ecc089aab5d" exitCode=0
Nov 03 21:14:36.866116 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:36.865991    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerDied","Data":"a34f00d7a6763d6be63467c7f34595921056010afa4a8cba99071ecc089aab5d"}
Nov 03 21:14:37.675002 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:37.674960    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:37.675002 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:37.675001    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:37.675211 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:37.675105    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:37.675245 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:37.675212    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:37.871683 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:37.871642    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" event={"ID":"5718e9a0-0924-41fe-a482-557054019ca9","Type":"ContainerStarted","Data":"89a0dd7e07f227e5e258a3aa8277a2b30686b81254d00956d747e0d4ffce7340"}
Nov 03 21:14:38.449507 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:38.449442    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:38.449691 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:38.449639    2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:14:38.450084 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:38.450069    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:14:38.560971 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:38.560923    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fp2r2" podStartSLOduration=8.360225053 podStartE2EDuration="21.560907318s" totalImagesPullingTime="13.200682265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:37.986732652 +0000 UTC m=+22.836431712" watchObservedRunningTime="2026-11-03 21:14:38.560907318 +0000 UTC m=+23.410606323"
Nov 03 21:14:38.675906 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:38.675691    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:38.675906 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:38.675851    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:38.877118 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:38.877013    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" event={"ID":"ae80c711-2bb7-43e4-9bf2-521dbc9e4291","Type":"ContainerStarted","Data":"3f233d061f21fea1bbc446adb4c25ea5e01a0794068764245434e9f6f8c258af"}
Nov 03 21:14:39.045285 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.045236    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8" podStartSLOduration=6.067867893 podStartE2EDuration="22.045222215s" totalImagesPullingTime="15.977354322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:39.044803672 +0000 UTC m=+23.894502713" watchObservedRunningTime="2026-11-03 21:14:39.045222215 +0000 UTC m=+23.894921220"
Nov 03 21:14:39.675345 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.675298    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:39.675525 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:39.675419    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:39.675525 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.675467    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:39.675606 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:39.675545    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:39.879802 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.879563    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:39.880188 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.879864    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:39.880188 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.879898    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:39.895604 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.895577    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:39.896073 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:39.896054    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:14:40.675546 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:40.675503    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:40.675750 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:40.675650    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:41.345315 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:41.345269    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s77gw"]
Nov 03 21:14:41.346202 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:41.345817    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:41.346202 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:41.345941    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:41.346452 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:41.346428    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kgjns"]
Nov 03 21:14:41.346564 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:41.346550    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:41.346674 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:41.346655    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:41.348135 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:41.348108    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tjq5q"]
Nov 03 21:14:41.348268 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:41.348214    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:41.348363 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:41.348322    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:42.675594 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:42.675555    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:42.676054 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:42.675562    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:42.676054 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:42.675663    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:42.676054 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:42.675760    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:43.675961 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:43.675873    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:43.676368 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:43.675984    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:44.675460 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:44.675411    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:44.675738 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:44.675437    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:44.675738 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:44.675535    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:44.675738 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:44.675616    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:45.675989 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:45.675949    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:45.676504 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:45.676044    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:46.675453 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:46.675419    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:46.675453 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:46.675454    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:46.675702 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:46.675525    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:46.675702 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:46.675591    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:47.535018 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:47.534987    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g7cv4_56dd4c25-1333-40f6-867d-1f02ff44b0d9/dns-node-resolver/0.log"
Nov 03 21:14:47.675640 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:47.675604    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:47.675798 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:47.675763    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:48.372187 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:48.372157    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pshfv_0d1e36e3-40f2-4c43-a009-e323301c57e6/node-ca/0.log"
Nov 03 21:14:48.676078 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:48.675998    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:48.676078 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:48.676018    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:48.676602 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:48.676123    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:48.676602 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:48.676239    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:49.405656 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:49.405611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:49.405821 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.405738    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:49.405821 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.405791    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret podName:8541995b-77c8-4973-9736-f750edf753cb nodeName:}" failed. No retries permitted until 2026-11-03 21:15:21.405779588 +0000 UTC m=+66.255478572 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret") pod "global-pull-secret-syncer-s77gw" (UID: "8541995b-77c8-4973-9736-f750edf753cb") : object "kube-system"/"original-pull-secret" not registered
Nov 03 21:14:49.506798 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:49.506758    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:49.506798 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:49.506801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:49.507047 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.506925    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 21:14:49.507047 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.506923    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:49.507047 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.506942    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 21:14:49.507047 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.506955    2568 projected.go:196] Error preparing data for projected volume kube-api-access-skx9b for pod openshift-network-diagnostics/network-check-target-kgjns: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:49.507047 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.506995    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs podName:be40fcfd-b776-4ede-ad34-e4be87a007e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:21.506980231 +0000 UTC m=+66.356679227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs") pod "network-metrics-daemon-tjq5q" (UID: "be40fcfd-b776-4ede-ad34-e4be87a007e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 21:14:49.507047 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.507012    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b podName:3408dd67-d555-4659-b48e-2366768b05c8 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:21.507004376 +0000 UTC m=+66.356703364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-skx9b" (UniqueName: "kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b") pod "network-check-target-kgjns" (UID: "3408dd67-d555-4659-b48e-2366768b05c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 21:14:49.675515 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:49.675438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:49.675663 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:49.675554    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:50.675470 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:50.675433    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:50.675934 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:50.675564    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:50.675934 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:50.675609    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:50.675934 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:50.675700    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:51.675456 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:51.675419    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:51.675657 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:51.675567    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:52.675398 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:52.675357    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:52.675398 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:52.675404    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:52.675601 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:52.675482    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:52.675601 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:52.675545    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:53.675771 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:53.675737    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:53.676214 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:53.675876    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:54.675229 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:54.675193    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:54.675426 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:54.675193    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:54.675426 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:54.675306    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:54.675426 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:54.675397    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:55.676496 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:55.676462    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:55.676951 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:55.676579    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:56.675301 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:56.675252    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:56.675301 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:56.675279    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:56.675583 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:56.675406    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:56.675583 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:56.675518    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:57.675883 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:57.675849    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:57.676260 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:57.675965    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:14:58.675819 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:58.675775    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:14:58.675983 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:58.675802    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:14:58.675983 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:58.675907    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:14:58.675983 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:58.675960    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:14:59.675046 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:14:59.675012    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:14:59.675219 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:14:59.675136    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:15:00.675012 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:00.674964    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:00.675436 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:00.675087    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:15:00.675436 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:00.675126    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:00.675436 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:00.675187    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:15:01.675827 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:01.675791    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:01.676229 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:01.675902    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:15:02.675928 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:02.675896    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:02.675928 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:02.675927    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:02.676399 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:02.675996    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:15:02.676399 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:02.676106    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:15:03.675982 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:03.675947    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:03.676406 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:03.676066    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:15:04.675244 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:04.675211    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:04.675469 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:04.675211    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:04.675469 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:04.675323    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgjns" podUID="3408dd67-d555-4659-b48e-2366768b05c8"
Nov 03 21:15:04.675469 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:04.675398    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s77gw" podUID="8541995b-77c8-4973-9736-f750edf753cb"
Nov 03 21:15:05.678231 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:05.677933    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:05.678231 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:05.678058    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjq5q" podUID="be40fcfd-b776-4ede-ad34-e4be87a007e0"
Nov 03 21:15:05.925114 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:05.925079    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeReady"
Nov 03 21:15:05.925315 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:05.925235    2568 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 21:15:06.018934 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.018906    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6jq4g"]
Nov 03 21:15:06.020690 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.020667    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.022134 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.022109    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fssfw"]
Nov 03 21:15:06.023764 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.023745    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.056230 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.056199    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cvcz7"]
Nov 03 21:15:06.058216 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.058201    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.072275 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.072254    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 21:15:06.072548 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.072534    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 21:15:06.072611 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.072535    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8l7\""
Nov 03 21:15:06.075062 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.075047    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rf5\""
Nov 03 21:15:06.075208 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.075194    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 21:15:06.075443 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.075429    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 21:15:06.075499 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.075469    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 21:15:06.083266 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.083242    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6jq4g"]
Nov 03 21:15:06.086272 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.086247    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fssfw"]
Nov 03 21:15:06.121591 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.121550    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dk7\" (UniqueName: \"kubernetes.io/projected/01063714-6865-449c-9726-caccbd2bfcc8-kube-api-access-m5dk7\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.121783 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.121601    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg44m\" (UniqueName: \"kubernetes.io/projected/1424c53b-1fc3-4c31-841d-f2db4741152a-kube-api-access-fg44m\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.121783 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.121638    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01063714-6865-449c-9726-caccbd2bfcc8-config-volume\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.121783 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.121662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01063714-6865-449c-9726-caccbd2bfcc8-metrics-tls\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.121783 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.121738    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1424c53b-1fc3-4c31-841d-f2db4741152a-cert\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.121783 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.121780    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01063714-6865-449c-9726-caccbd2bfcc8-tmp-dir\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.148963 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.148917    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvcz7"]
Nov 03 21:15:06.172589 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.172540    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tq7pf\""
Nov 03 21:15:06.172589 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.172570    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 21:15:06.172589 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.172574    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 21:15:06.172589 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.172592    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 21:15:06.172851 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.172606    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 21:15:06.223094 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrj7\" (UniqueName: \"kubernetes.io/projected/e7533013-9173-474b-9baf-1da837310ea1-kube-api-access-dgrj7\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.223190 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223108    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7533013-9173-474b-9baf-1da837310ea1-data-volume\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.223190 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223133    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dk7\" (UniqueName: \"kubernetes.io/projected/01063714-6865-449c-9726-caccbd2bfcc8-kube-api-access-m5dk7\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.223287 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223204    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg44m\" (UniqueName: \"kubernetes.io/projected/1424c53b-1fc3-4c31-841d-f2db4741152a-kube-api-access-fg44m\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.223350 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223320    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7533013-9173-474b-9baf-1da837310ea1-crio-socket\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.223461 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01063714-6865-449c-9726-caccbd2bfcc8-config-volume\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.223535 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223477    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01063714-6865-449c-9726-caccbd2bfcc8-metrics-tls\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.223535 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223513    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1424c53b-1fc3-4c31-841d-f2db4741152a-cert\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.223642 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223550    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7533013-9173-474b-9baf-1da837310ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.223642 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7533013-9173-474b-9baf-1da837310ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.223642 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223617    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01063714-6865-449c-9726-caccbd2bfcc8-tmp-dir\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.223953 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.223934    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01063714-6865-449c-9726-caccbd2bfcc8-tmp-dir\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.224084 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.224066    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01063714-6865-449c-9726-caccbd2bfcc8-config-volume\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.227180 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.227156    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01063714-6865-449c-9726-caccbd2bfcc8-metrics-tls\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.227287 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.227190    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1424c53b-1fc3-4c31-841d-f2db4741152a-cert\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.281848 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.281779    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dk7\" (UniqueName: \"kubernetes.io/projected/01063714-6865-449c-9726-caccbd2bfcc8-kube-api-access-m5dk7\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.281848 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.281828    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg44m\" (UniqueName: \"kubernetes.io/projected/1424c53b-1fc3-4c31-841d-f2db4741152a-kube-api-access-fg44m\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.324840 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.324806    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrj7\" (UniqueName: \"kubernetes.io/projected/e7533013-9173-474b-9baf-1da837310ea1-kube-api-access-dgrj7\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.324993 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.324852    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7533013-9173-474b-9baf-1da837310ea1-data-volume\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.324993 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.324882    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7533013-9173-474b-9baf-1da837310ea1-crio-socket\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.324993 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.324912    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7533013-9173-474b-9baf-1da837310ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.324993 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.324927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7533013-9173-474b-9baf-1da837310ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.325151 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.325115    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7533013-9173-474b-9baf-1da837310ea1-crio-socket\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.325318 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.325302    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7533013-9173-474b-9baf-1da837310ea1-data-volume\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.325552 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.325524    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7533013-9173-474b-9baf-1da837310ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.327366 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.327342    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7533013-9173-474b-9baf-1da837310ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.333802 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.333785    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:06.337960 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.337942    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:06.381295 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.381263    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrj7\" (UniqueName: \"kubernetes.io/projected/e7533013-9173-474b-9baf-1da837310ea1-kube-api-access-dgrj7\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.669695 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.669616    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:06.675518 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.675495    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:06.675617 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.675502    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:06.715799 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.715563    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fssfw"]
Nov 03 21:15:06.728285 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.727803    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 21:15:06.728285 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.727812    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 21:15:06.728285 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.727916    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\""
Nov 03 21:15:06.728285 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.728162    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 21:15:06.728285 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:06.728164    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1424c53b_1fc3_4c31_841d_f2db4741152a.slice/crio-0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0 WatchSource:0}: Error finding container 0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0: Status 404 returned error can't find the container with id 0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0
Nov 03 21:15:06.728285 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.728175    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\""
Nov 03 21:15:06.766426 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.766311    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6jq4g"]
Nov 03 21:15:06.770497 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:06.770454    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01063714_6865_449c_9726_caccbd2bfcc8.slice/crio-38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55 WatchSource:0}: Error finding container 38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55: Status 404 returned error can't find the container with id 38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55
Nov 03 21:15:06.940004 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.939970    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fssfw" event={"ID":"1424c53b-1fc3-4c31-841d-f2db4741152a","Type":"ContainerStarted","Data":"0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0"}
Nov 03 21:15:06.941089 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:06.941065    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6jq4g" event={"ID":"01063714-6865-449c-9726-caccbd2bfcc8","Type":"ContainerStarted","Data":"38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55"}
Nov 03 21:15:07.037846 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:07.037813    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvcz7"]
Nov 03 21:15:07.041295 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:07.041270    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7533013_9173_474b_9baf_1da837310ea1.slice/crio-33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef WatchSource:0}: Error finding container 33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef: Status 404 returned error can't find the container with id 33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef
Nov 03 21:15:07.679555 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:07.679520    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:07.731620 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:07.731588    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\""
Nov 03 21:15:07.732034 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:07.731675    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 21:15:07.946153 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:07.946119    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvcz7" event={"ID":"e7533013-9173-474b-9baf-1da837310ea1","Type":"ContainerStarted","Data":"09c360b9ec0238176bf32d7ae1cde857270e35d5ea3279625b0959bc5c777273"}
Nov 03 21:15:07.946430 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:07.946192    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvcz7" event={"ID":"e7533013-9173-474b-9baf-1da837310ea1","Type":"ContainerStarted","Data":"33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef"}
Nov 03 21:15:08.606672 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.606589    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"]
Nov 03 21:15:08.615110 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.615079    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.667200 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.667164    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"]
Nov 03 21:15:08.715615 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.715578    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 21:15:08.716349 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.716305    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 21:15:08.716462 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.716408    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 21:15:08.717298 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.717112    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 21:15:08.717298 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.717157    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 21:15:08.717298 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.717254    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wqd2s\""
Nov 03 21:15:08.745605 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.745577    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.746044 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.745679    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jw65\" (UniqueName: \"kubernetes.io/projected/67b25889-3fc6-4a28-99fa-267620698905-kube-api-access-7jw65\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.746044 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.745699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b25889-3fc6-4a28-99fa-267620698905-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.746044 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.745720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.847190 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.847153    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw65\" (UniqueName: \"kubernetes.io/projected/67b25889-3fc6-4a28-99fa-267620698905-kube-api-access-7jw65\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.847399 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.847203    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b25889-3fc6-4a28-99fa-267620698905-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.847399 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.847237    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.847399 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.847269    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.848678 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.848629    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b25889-3fc6-4a28-99fa-267620698905-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.850790 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.850762    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.850899 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.850769    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.904525 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.904457    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw65\" (UniqueName: \"kubernetes.io/projected/67b25889-3fc6-4a28-99fa-267620698905-kube-api-access-7jw65\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.929172 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.929136    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:08.951524 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:08.951496    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvcz7" event={"ID":"e7533013-9173-474b-9baf-1da837310ea1","Type":"ContainerStarted","Data":"6696722c8fdd01f121d2db2cf547ce2418c78b84f3d9501b9a15e1bc4136f59a"}
Nov 03 21:15:09.315392 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:09.315240    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"]
Nov 03 21:15:09.353469 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:09.353436    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b25889_3fc6_4a28_99fa_267620698905.slice/crio-10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413 WatchSource:0}: Error finding container 10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413: Status 404 returned error can't find the container with id 10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413
Nov 03 21:15:09.956272 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:09.956239    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2" event={"ID":"67b25889-3fc6-4a28-99fa-267620698905","Type":"ContainerStarted","Data":"10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413"}
Nov 03 21:15:09.958211 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:09.958180    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6jq4g" event={"ID":"01063714-6865-449c-9726-caccbd2bfcc8","Type":"ContainerStarted","Data":"b511a928c80b040710dfc6022093316460d758801dd044f949581cc8d2d7934b"}
Nov 03 21:15:09.958309 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:09.958228    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6jq4g" event={"ID":"01063714-6865-449c-9726-caccbd2bfcc8","Type":"ContainerStarted","Data":"4c8a8b7da990632132756f494f3f52e154e19636d8bde911a988b63438e493d5"}
Nov 03 21:15:09.958381 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:09.958344    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:10.072183 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:10.071706    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6jq4g" podStartSLOduration=2.904076862 podStartE2EDuration="5.07168666s" totalImagesPullingTime="2.167609798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:10.070568355 +0000 UTC m=+54.920267373" watchObservedRunningTime="2026-11-03 21:15:10.07168666 +0000 UTC m=+54.921385667"
Nov 03 21:15:10.965603 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:10.965571    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvcz7" event={"ID":"e7533013-9173-474b-9baf-1da837310ea1","Type":"ContainerStarted","Data":"6e62b12a4331be78564e302a45a90c96e5c729cb629813cbf26d1cbe4c475fa2"}
Nov 03 21:15:10.967547 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:10.967521    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fssfw" event={"ID":"1424c53b-1fc3-4c31-841d-f2db4741152a","Type":"ContainerStarted","Data":"3b24d97b3d6a06f92dbd7350294012b0e85e00a55e5ad562e08b61cb3201e3eb"}
Nov 03 21:15:11.088227 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:11.087202    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cvcz7" podStartSLOduration=2.163898377 podStartE2EDuration="5.087182303s" totalImagesPullingTime="2.923283926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:15:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:11.085307777 +0000 UTC m=+55.935006784" watchObservedRunningTime="2026-11-03 21:15:11.087182303 +0000 UTC m=+55.936881310"
Nov 03 21:15:11.200322 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:11.200268    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fssfw" podStartSLOduration=3.01165125 podStartE2EDuration="6.200253288s" totalImagesPullingTime="3.188602038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:11.198783723 +0000 UTC m=+56.048482730" watchObservedRunningTime="2026-11-03 21:15:11.200253288 +0000 UTC m=+56.049952294"
Nov 03 21:15:11.895575 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:11.895544    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:11.973990 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:11.973954    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2" event={"ID":"67b25889-3fc6-4a28-99fa-267620698905","Type":"ContainerStarted","Data":"9b6ac280f3ea8339e10283f7f3e07bc826aed6c9286cc2319c5e6b99e104cb9f"}
Nov 03 21:15:11.974466 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:11.974013    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2" event={"ID":"67b25889-3fc6-4a28-99fa-267620698905","Type":"ContainerStarted","Data":"c6bebe826be2d72d6bd3f3345ce7c024ee3dfadef43810b072c9052cfb2a7e55"}
Nov 03 21:15:12.281487 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:12.281421    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2" podStartSLOduration=2.732802564 podStartE2EDuration="4.281373572s" totalImagesPullingTime="1.548571008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:12.281126608 +0000 UTC m=+57.130825614" watchObservedRunningTime="2026-11-03 21:15:12.281373572 +0000 UTC m=+57.131072578"
Nov 03 21:15:15.296845 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.296813    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xr2gw"]
Nov 03 21:15:15.298869 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.298853    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.360380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.360319    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 21:15:15.360380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.360320    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79p29\""
Nov 03 21:15:15.394981 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.394943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395112 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.394986    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-tls\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395112 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395016    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlp9\" (UniqueName: \"kubernetes.io/projected/274de09e-a693-46cf-b317-a0cbdfcf79df-kube-api-access-fzlp9\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395108    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-wtmp\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395193 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395169    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395282 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395202    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-metrics-client-ca\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395282 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395220    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-sys\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395282 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-root\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.395443 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.395362    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-textfile\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.400487 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.400466    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 21:15:15.425471 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.425438    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 21:15:15.495863 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495800    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-metrics-client-ca\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.495863 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495865    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-sys\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495887    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-root\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495914    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-textfile\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495943    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-root\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495958    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495944    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-sys\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.495992    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-tls\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496010    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlp9\" (UniqueName: \"kubernetes.io/projected/274de09e-a693-46cf-b317-a0cbdfcf79df-kube-api-access-fzlp9\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-wtmp\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496105 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496492 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496216    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-wtmp\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496492 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496356    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-textfile\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496575 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496558    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-metrics-client-ca\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.496609 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.496574    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.498618 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.498601    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.498706 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.498691    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-tls\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.558048 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.557982    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlp9\" (UniqueName: \"kubernetes.io/projected/274de09e-a693-46cf-b317-a0cbdfcf79df-kube-api-access-fzlp9\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.661177 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.661138    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79p29\""
Nov 03 21:15:15.671202 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.671171    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:15.681363 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:15.681311    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274de09e_a693_46cf_b317_a0cbdfcf79df.slice/crio-df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3 WatchSource:0}: Error finding container df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3: Status 404 returned error can't find the container with id df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3
Nov 03 21:15:15.994120 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:15.994088    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr2gw" event={"ID":"274de09e-a693-46cf-b317-a0cbdfcf79df","Type":"ContainerStarted","Data":"df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3"}
Nov 03 21:15:18.001891 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:18.001859    2568 generic.go:356] "Generic (PLEG): container finished" podID="274de09e-a693-46cf-b317-a0cbdfcf79df" containerID="12fa484184466184676e48761907402b24f8fa45cc853ef85d9c38fa54a7540d" exitCode=0
Nov 03 21:15:18.002278 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:18.001929    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr2gw" event={"ID":"274de09e-a693-46cf-b317-a0cbdfcf79df","Type":"ContainerDied","Data":"12fa484184466184676e48761907402b24f8fa45cc853ef85d9c38fa54a7540d"}
Nov 03 21:15:19.007842 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:19.007805    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr2gw" event={"ID":"274de09e-a693-46cf-b317-a0cbdfcf79df","Type":"ContainerStarted","Data":"3ac34f90487c2938d267a5a4e8b163bc772635347e46a975ae1a304cf03d388a"}
Nov 03 21:15:19.008229 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:19.007859    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xr2gw" event={"ID":"274de09e-a693-46cf-b317-a0cbdfcf79df","Type":"ContainerStarted","Data":"30c2f0e1d89026d09bbe17ed808cf69442b68379d709e62461575301fe7e1f30"}
Nov 03 21:15:19.125423 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:19.125373    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xr2gw" podStartSLOduration=2.786999953 podStartE2EDuration="4.125359246s" totalImagesPullingTime="1.261630337s" totalInitContainerRuntime="76.728956ms" isStatefulPod=true podCreationTimestamp="2026-11-03 21:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:19.125307872 +0000 UTC m=+63.975006885" watchObservedRunningTime="2026-11-03 21:15:19.125359246 +0000 UTC m=+63.975058245"
Nov 03 21:15:19.949666 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:19.949640    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"]
Nov 03 21:15:19.951377 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:19.951362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:19.970149 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:19.970125    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:20.002734 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.002700    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 21:15:20.002734 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.002720    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m96vh\""
Nov 03 21:15:20.010514 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.010490    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"]
Nov 03 21:15:20.031625 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.031600    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdtj\" (UniqueName: \"kubernetes.io/projected/62d2b800-0c2b-49ca-906e-ec29326e1726-kube-api-access-5wdtj\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.031856 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.031824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.132292 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.132251    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.132504 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.132302    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdtj\" (UniqueName: \"kubernetes.io/projected/62d2b800-0c2b-49ca-906e-ec29326e1726-kube-api-access-5wdtj\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.132504 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:20.132418    2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Nov 03 21:15:20.132504 ip-10-0-129-65 kubenswrapper[2568]: E1103 21:15:20.132495    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert podName:62d2b800-0c2b-49ca-906e-ec29326e1726 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:20.63247916 +0000 UTC m=+65.482178157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-gv74f" (UID: "62d2b800-0c2b-49ca-906e-ec29326e1726") : secret "monitoring-plugin-cert" not found
Nov 03 21:15:20.191568 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.191532    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdtj\" (UniqueName: \"kubernetes.io/projected/62d2b800-0c2b-49ca-906e-ec29326e1726-kube-api-access-5wdtj\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.612249 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.612214    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78789694d8-xqjq4"]
Nov 03 21:15:20.614077 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.614061    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.637866 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.637833    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.637866 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.637867    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.638063 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.637894    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.638063 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.638000    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.638063 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.638036    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.638179 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.638071    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.638179 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.638171    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.638246 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.638187    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.638246 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.638227    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.640336 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.640310    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.666680 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.666651    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 21:15:20.666842 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.666796    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kz2rr\""
Nov 03 21:15:20.666842 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.666818    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 21:15:20.667112 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.667098    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 21:15:20.716392 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.716314    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 21:15:20.716793 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.716544    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 21:15:20.716793 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.716562    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 21:15:20.716793 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.716565    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 21:15:20.739206 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739175    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739209    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739300    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739342    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739380 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739634 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739433    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739634 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.739912 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.739878    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.740203 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.740182    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.740271 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.740165    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.740366 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.740219    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.742025 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.741991    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.742100 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.742023    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.767100 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.767061    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 21:15:20.770581 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.770558    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.817021 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.816980    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78789694d8-xqjq4"]
Nov 03 21:15:20.818652 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.818632    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.861555 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.861492    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:20.924471 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.924445    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:20.988207 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.988140    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-964cff64-qf6zt"]
Nov 03 21:15:20.990809 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:20.990774    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042241 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042215    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc76r\" (UniqueName: \"kubernetes.io/projected/6679db18-7f2c-4478-9eb1-0a20b16040e4-kube-api-access-qc76r\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042258    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-serving-certs-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042404    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042431    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042511    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042532    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-federate-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042551    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.042645 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.042615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-metrics-client-ca\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.043297 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.043182    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 21:15:21.043297 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.043237    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 21:15:21.043297 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.043270    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 21:15:21.043297 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.043290    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 21:15:21.043579 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.043359    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 21:15:21.043669 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.043653    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rdfhn\""
Nov 03 21:15:21.044186 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.044172    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 21:15:21.053956 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.053929    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-964cff64-qf6zt"]
Nov 03 21:15:21.143352 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143241    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143352 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143276    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143352 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143300    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143352 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143319    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-federate-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143693 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143370    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143693 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-metrics-client-ca\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143693 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143585    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc76r\" (UniqueName: \"kubernetes.io/projected/6679db18-7f2c-4478-9eb1-0a20b16040e4-kube-api-access-qc76r\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.143693 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.143628    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-serving-certs-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.144447 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.144425    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.144910 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.144887    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-serving-certs-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.144954 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.144914    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-metrics-client-ca\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.146144 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.146125    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-federate-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.146230 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.146150    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.146431 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.146415    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.146494 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.146476    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.200070 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.200040    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc76r\" (UniqueName: \"kubernetes.io/projected/6679db18-7f2c-4478-9eb1-0a20b16040e4-kube-api-access-qc76r\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.225590 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.225558    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"]
Nov 03 21:15:21.228447 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:21.228422    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d2b800_0c2b_49ca_906e_ec29326e1726.slice/crio-6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2 WatchSource:0}: Error finding container 6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2: Status 404 returned error can't find the container with id 6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2
Nov 03 21:15:21.294261 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.294226    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78789694d8-xqjq4"]
Nov 03 21:15:21.298628 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:21.298601    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode327fb0c_d657_4d21_862d_de5f5fd7dbd3.slice/crio-4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53 WatchSource:0}: Error finding container 4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53: Status 404 returned error can't find the container with id 4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53
Nov 03 21:15:21.303043 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.303024    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:21.446764 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.446730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:21.497586 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.497552    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 21:15:21.509927 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.509898    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:21.547417 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.547384    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:21.547531 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.547447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:21.598253 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.598209    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 21:15:21.598415 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.598384    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 21:15:21.610777 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.610747    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:21.658980 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.658951    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 21:15:21.671519 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.671488    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:21.723942 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.723871    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-964cff64-qf6zt"]
Nov 03 21:15:21.727118 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:21.727078    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6679db18_7f2c_4478_9eb1_0a20b16040e4.slice/crio-48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136 WatchSource:0}: Error finding container 48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136: Status 404 returned error can't find the container with id 48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136
Nov 03 21:15:21.737068 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.737045    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\""
Nov 03 21:15:21.742205 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.742187    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\""
Nov 03 21:15:21.747359 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.747342    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:21.751687 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.751665    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:21.844966 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.844942    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\""
Nov 03 21:15:21.855183 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:21.855164    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:22.019469 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:22.019365    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt" event={"ID":"6679db18-7f2c-4478-9eb1-0a20b16040e4","Type":"ContainerStarted","Data":"48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136"}
Nov 03 21:15:22.020704 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:22.020677    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78789694d8-xqjq4" event={"ID":"e327fb0c-d657-4d21-862d-de5f5fd7dbd3","Type":"ContainerStarted","Data":"4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53"}
Nov 03 21:15:22.021822 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:22.021799    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f" event={"ID":"62d2b800-0c2b-49ca-906e-ec29326e1726","Type":"ContainerStarted","Data":"6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2"}
Nov 03 21:15:22.123894 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:22.123864    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kgjns"]
Nov 03 21:15:22.124865 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:22.124841    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s77gw"]
Nov 03 21:15:22.127124 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:22.127096    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8541995b_77c8_4973_9736_f750edf753cb.slice/crio-6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557 WatchSource:0}: Error finding container 6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557: Status 404 returned error can't find the container with id 6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557
Nov 03 21:15:22.227791 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:22.227756    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tjq5q"]
Nov 03 21:15:22.232778 ip-10-0-129-65 kubenswrapper[2568]: W1103 21:15:22.232738    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe40fcfd_b776_4ede_ad34_e4be87a007e0.slice/crio-765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504 WatchSource:0}: Error finding container 765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504: Status 404 returned error can't find the container with id 765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504
Nov 03 21:15:23.026924 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:23.026879    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kgjns" event={"ID":"3408dd67-d555-4659-b48e-2366768b05c8","Type":"ContainerStarted","Data":"6cfa1bf55155702b4e2b5e97acb4c607054f8fb029d8863e2099b9b92aec39a0"}
Nov 03 21:15:23.028400 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:23.028368    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjq5q" event={"ID":"be40fcfd-b776-4ede-ad34-e4be87a007e0","Type":"ContainerStarted","Data":"765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504"}
Nov 03 21:15:23.030010 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:23.029982    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s77gw" event={"ID":"8541995b-77c8-4973-9736-f750edf753cb","Type":"ContainerStarted","Data":"6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557"}
Nov 03 21:15:24.036734 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:24.036693    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f" event={"ID":"62d2b800-0c2b-49ca-906e-ec29326e1726","Type":"ContainerStarted","Data":"9eb8f9e0c5eab96be592a59d44752487818ceae6da95b275e26e492bae47eaaa"}
Nov 03 21:15:24.037834 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:24.037776    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:24.047366 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:24.047322    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:24.269207 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:24.269141    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f" podStartSLOduration=3.44775015 podStartE2EDuration="5.268991676s" totalImagesPullingTime="1.821241526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:24.151838233 +0000 UTC m=+69.001537232" watchObservedRunningTime="2026-11-03 21:15:24.268991676 +0000 UTC m=+69.118690683"
Nov 03 21:15:25.043073 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:25.043038    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt" event={"ID":"6679db18-7f2c-4478-9eb1-0a20b16040e4","Type":"ContainerStarted","Data":"4a5dc01ee0784372bc77dd5adc3a7471d2f9a15a0bd50f38e330ee30c00b8d59"}
Nov 03 21:15:25.046274 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:25.046239    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjq5q" event={"ID":"be40fcfd-b776-4ede-ad34-e4be87a007e0","Type":"ContainerStarted","Data":"bd09ae993faaf2b8cf2f27e32b8cc1704a675802cda1a43351b712cf298c2580"}
Nov 03 21:15:25.046442 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:25.046297    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjq5q" event={"ID":"be40fcfd-b776-4ede-ad34-e4be87a007e0","Type":"ContainerStarted","Data":"3304d25a2f96c04abe7ae3797d9067951e6ba5fcd42a3017ba3c3e174ef97fad"}
Nov 03 21:15:27.060677 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:27.060640    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt" event={"ID":"6679db18-7f2c-4478-9eb1-0a20b16040e4","Type":"ContainerStarted","Data":"3df5c1335b0386b89f04c9affda73ee26ce8d596d20bc913cc9a0252be3c0eaf"}
Nov 03 21:15:27.061083 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:27.060718    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt" event={"ID":"6679db18-7f2c-4478-9eb1-0a20b16040e4","Type":"ContainerStarted","Data":"b97bfd962caced40581da40d84aedec0475d4f06e94aab0f0cbca701ec98fafc"}
Nov 03 21:15:27.063075 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:27.063043    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78789694d8-xqjq4" event={"ID":"e327fb0c-d657-4d21-862d-de5f5fd7dbd3","Type":"ContainerStarted","Data":"191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5"}
Nov 03 21:15:27.334096 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:27.333988    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-964cff64-qf6zt" podStartSLOduration=2.61845595 podStartE2EDuration="7.333969027s" totalImagesPullingTime="4.715513077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:27.284384684 +0000 UTC m=+72.134083690" watchObservedRunningTime="2026-11-03 21:15:27.333969027 +0000 UTC m=+72.183668035"
Nov 03 21:15:27.335296 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:27.335245    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tjq5q" podStartSLOduration=68.390413308 podStartE2EDuration="1m10.335231938s" totalImagesPullingTime="1.94481863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:25.160693315 +0000 UTC m=+70.010392321" watchObservedRunningTime="2026-11-03 21:15:27.335231938 +0000 UTC m=+72.184930947"
Nov 03 21:15:27.449826 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:27.449768    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78789694d8-xqjq4" podStartSLOduration=2.77692976 podStartE2EDuration="7.449750634s" totalImagesPullingTime="4.672820874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:27.448313586 +0000 UTC m=+72.298012593" watchObservedRunningTime="2026-11-03 21:15:27.449750634 +0000 UTC m=+72.299449640"
Nov 03 21:15:28.068978 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:28.068935    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kgjns" event={"ID":"3408dd67-d555-4659-b48e-2366768b05c8","Type":"ContainerStarted","Data":"126ee0edc95c7b8cd8ee540e814b70bba0f9a2ef2861855eeee500ac398a5c4f"}
Nov 03 21:15:28.069488 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:28.069433    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:28.201431 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:28.201365    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kgjns" podStartSLOduration=66.284747806 podStartE2EDuration="1m11.201347039s" totalImagesPullingTime="4.916599233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:28.19922334 +0000 UTC m=+73.048922350" watchObservedRunningTime="2026-11-03 21:15:28.201347039 +0000 UTC m=+73.051046047"
Nov 03 21:15:28.441222 ip-10-0-129-65 kubenswrapper[2568]: I1103 21:15:28.441184    2568 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78789694d8-xqjq4"]
Nov 03 21:15:28.937730 ip-10-0-129-65 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 21:15:28.953810 ip-10-0-129-65 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 21:15:28.954040 ip-10-0-129-65 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 21:15:28.954106 ip-10-0-129-65 systemd[1]: kubelet.service: Consumed 3.976s CPU time.
Nov 03 21:15:28.962212 ip-10-0-129-65 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 21:15:29.049388 ip-10-0-129-65 kubenswrapper[5244]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:15:29.049388 ip-10-0-129-65 kubenswrapper[5244]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 21:15:29.049388 ip-10-0-129-65 kubenswrapper[5244]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:15:29.049388 ip-10-0-129-65 kubenswrapper[5244]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 21:15:29.051462 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051378    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.051462 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051460    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.051462 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051465    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051469    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051472    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051475    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051478    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051481    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051484    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051486    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051489    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051491    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051494    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051496    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051499    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051502    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051505    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051507    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051510    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051512    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051515    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.051553 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051518    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051521    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051523    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051526    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051528    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051531    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051533    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051542    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051545    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051547    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051550    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051553    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051558    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051562    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051565    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051568    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051571    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051574    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051577    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.052064 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051579    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051582    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051584    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051587    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051590    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051593    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051597    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051601    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051603    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051606    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051608    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051610    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051613    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051615    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051618    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051622    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051626    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051629    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051633    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051636    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.052771 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051641    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051645    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051649    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051652    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051656    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051660    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051665    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051669    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051673    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051677    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051681    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051685    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051689    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051693    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051695    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051698    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051700    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051703    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051706    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051708    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.053591 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051712    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051714    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051717    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051719    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051722    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051724    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051729    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051732    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051735    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051737    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051740    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051743    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051747    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051752    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051756    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051772    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051776    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051778    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051781    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.054363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051783    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051786    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051788    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051791    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051795    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051797    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.051800    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053881    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053893    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053897    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053901    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053905    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053910    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053914    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053919    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053923    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053927    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053931    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053935    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.054955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053939    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053943    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053947    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053951    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053955    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053959    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053962    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053967    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053971    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053976    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053980    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053986    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053990    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053993    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.053998    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054002    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054006    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054011    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054016    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.055495 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054022    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054026    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054030    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054034    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054037    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054041    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054047    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054053    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054057    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054062    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054067    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054070    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054074    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054078    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054082    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054086    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054090    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054093    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054098    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.055961 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054101    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054106    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054110    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054114    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054118    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054122    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054126    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054130    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054133    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054137    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054141    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054145    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054149    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054153    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054156    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054160    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054164    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054168    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054172    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.056468 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054176    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054179    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054185    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054189    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054194    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054198    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054202    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054207    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054211    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054215    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054219    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054222    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054226    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054230    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054234    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054239    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054243    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054248    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054252    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054256    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.056937 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054260    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054263    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054267    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054271    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054275    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054279    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054283    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054287    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054291    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054295    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054299    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054303    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054307    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054312    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054316    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054320    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054350    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054961    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054969    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.057458 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054972    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054975    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054979    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054982    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054984    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054987    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054989    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054992    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054995    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.054997    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055000    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055003    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055005    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055008    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055011    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055014    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055016    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055019    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055021    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055024    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.057917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055028    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055032    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055036    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055039    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055041    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055044    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055047    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055049    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055052    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055054    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055057    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055059    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055062    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055065    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055068    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055070    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055073    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055075    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055078    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055080    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.058429 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055083    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055085    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055088    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055091    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055094    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055096    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055098    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055101    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055103    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055106    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055109    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055112    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055114    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055117    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055119    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055122    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055125    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055127    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055130    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.058930 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055132    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055134    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055137    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055139    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055142    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055145    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055148    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055150    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055153    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055155    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055158    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055160    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055166    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055168    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055170    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055173    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055175    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055178    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055180    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055183    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.059785 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055186    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055188    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055192    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055195    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055198    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055202    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055205    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055207    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055210    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055213    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055215    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055218    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055220    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055223    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055225    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055228    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055230    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055235    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055237    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.060736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055240    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055242    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055245    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055247    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055250    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.055255    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055378    5244 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055395    5244 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055402    5244 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055407    5244 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055411    5244 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055414    5244 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055418    5244 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055422    5244 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055426    5244 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055429    5244 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055432    5244 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055436    5244 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055439    5244 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055442    5244 flags.go:64] FLAG: --cgroup-root=""
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055445    5244 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055448    5244 flags.go:64] FLAG: --client-ca-file=""
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055451    5244 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 21:15:29.061735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055454    5244 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055458    5244 flags.go:64] FLAG: --cluster-domain=""
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055460    5244 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055463    5244 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055467    5244 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055470    5244 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055474    5244 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055479    5244 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055482    5244 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055485    5244 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055488    5244 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055491    5244 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055494    5244 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055497    5244 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055502    5244 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055506    5244 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055509    5244 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055512    5244 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055515    5244 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055518    5244 flags.go:64] FLAG: --enable-server="true"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055521    5244 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055525    5244 flags.go:64] FLAG: --event-burst="100"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055528    5244 flags.go:64] FLAG: --event-qps="50"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055531    5244 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 21:15:29.062847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055534    5244 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055537    5244 flags.go:64] FLAG: --eviction-hard=""
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055541    5244 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055545    5244 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055548    5244 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055551    5244 flags.go:64] FLAG: --eviction-soft=""
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055553    5244 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055556    5244 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055559    5244 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055562    5244 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055565    5244 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055568    5244 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055570    5244 flags.go:64] FLAG: --feature-gates=""
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055574    5244 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055577    5244 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055581    5244 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055584    5244 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055588    5244 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055591    5244 flags.go:64] FLAG: --help="false"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055594    5244 flags.go:64] FLAG: --hostname-override="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055598    5244 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055600    5244 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055604    5244 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 21:15:29.063962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055608    5244 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055612    5244 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055615    5244 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055617    5244 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055620    5244 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055623    5244 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055626    5244 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055629    5244 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055632    5244 flags.go:64] FLAG: --kube-reserved=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055636    5244 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055639    5244 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055642    5244 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055645    5244 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055647    5244 flags.go:64] FLAG: --lock-file=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055650    5244 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055653    5244 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055656    5244 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055661    5244 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055664    5244 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055667    5244 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055670    5244 flags.go:64] FLAG: --logging-format="text"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055672    5244 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055676    5244 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055678    5244 flags.go:64] FLAG: --manifest-url=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055683    5244 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 21:15:29.064981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055688    5244 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055691    5244 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055699    5244 flags.go:64] FLAG: --max-pods="110"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055702    5244 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055705    5244 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055708    5244 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055712    5244 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055715    5244 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055718    5244 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055721    5244 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055728    5244 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055731    5244 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055734    5244 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055737    5244 flags.go:64] FLAG: --pod-cidr=""
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055739    5244 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055742    5244 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055745    5244 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055748    5244 flags.go:64] FLAG: --port="10250"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055751    5244 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055754    5244 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-023f3183d384f3e57"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055757    5244 flags.go:64] FLAG: --qos-reserved=""
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055760    5244 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055763    5244 flags.go:64] FLAG: --register-node="true"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055766    5244 flags.go:64] FLAG: --register-with-taints=""
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055769    5244 flags.go:64] FLAG: --registry-burst="10"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055772    5244 flags.go:64] FLAG: --registry-qps="5"
Nov 03 21:15:29.066073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055775    5244 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055778    5244 flags.go:64] FLAG: --reserved-memory=""
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055782    5244 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055785    5244 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055788    5244 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055792    5244 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055795    5244 flags.go:64] FLAG: --runonce="false"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055798    5244 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055801    5244 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055805    5244 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055808    5244 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055810    5244 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055815    5244 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055818    5244 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055821    5244 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055823    5244 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055826    5244 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055829    5244 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055832    5244 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055836    5244 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055838    5244 flags.go:64] FLAG: --system-cgroups=""
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055841    5244 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055846    5244 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055849    5244 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055852    5244 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 21:15:29.067206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055856    5244 flags.go:64] FLAG: --tls-min-version=""
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055858    5244 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055861    5244 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055864    5244 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055867    5244 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055870    5244 flags.go:64] FLAG: --v="2"
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055874    5244 flags.go:64] FLAG: --version="false"
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055878    5244 flags.go:64] FLAG: --vmodule=""
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055882    5244 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.055885    5244 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056015    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056020    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056025    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056027    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056031    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056034    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056036    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056042    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056045    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056048    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056050    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056053    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.068308 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056056    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056058    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056061    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056063    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056066    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056068    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056071    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056073    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056076    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056078    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056081    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056083    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056086    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056089    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056091    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056094    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056096    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056098    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056102    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.069252 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056106    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056109    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056112    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056116    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056118    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056121    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056123    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056126    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056128    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056131    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056134    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056137    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056139    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056141    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056144    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056146    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056149    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056151    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056153    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056156    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.069922 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056158    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056161    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056163    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056166    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056168    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056171    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056173    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056176    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056178    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056180    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056183    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056185    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056188    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056190    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056193    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056196    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056199    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056201    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056204    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.070558 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056206    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056208    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056211    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056214    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056216    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056220    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056223    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056226    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056228    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056231    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056233    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056236    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056238    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056241    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056243    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056245    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056248    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056250    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056253    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.071094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056255    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056257    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056260    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056262    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056265    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056267    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056269    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056272    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056274    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056278    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056280    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056283    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056285    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056288    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056290    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056292    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.071642 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056295    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.056320    5244 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056373    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056378    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056381    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056384    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056388    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056391    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056394    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056396    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056398    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056402    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056405    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056408    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056410    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056413    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.072083 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056415    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056418    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056421    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056423    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056425    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056428    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056430    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056433    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056435    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056439    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056442    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056444    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056447    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056449    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056452    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056454    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056457    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056459    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056462    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.072502 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056465    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056468    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056470    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056473    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056477    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056481    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056484    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056486    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056490    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056493    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056496    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056498    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056501    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056503    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056505    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056508    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056510    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056513    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056516    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.072965 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056518    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056521    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056523    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056526    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056529    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056531    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056533    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056536    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056538    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056541    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056543    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056546    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056548    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056551    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056554    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056556    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056559    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056561    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056564    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056566    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.073585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056568    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056572    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056574    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056577    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056580    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056582    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056585    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056589    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056591    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056594    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056597    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056600    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056602    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056605    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056607    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056610    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056612    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056615    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056617    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056620    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.074399 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056623    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056625    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056628    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056630    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056633    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056635    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056640    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056644    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056648    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056652    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056654    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056657    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056659    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.056664    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.061866    5244 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.061884    5244 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.061909    5244 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.061918    5244 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062022    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.074928 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062030    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062035    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062039    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062044    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062048    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062052    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062057    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062061    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062065    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062069    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062074    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062078    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062083    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062087    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062092    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062096    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062101    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062105    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062109    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062121    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.075405 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062126    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062131    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062135    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062141    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062145    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062151    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062158    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062163    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062167    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062172    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062176    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062181    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062185    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062189    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062193    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062197    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062201    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062206    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062210    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062214    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.075906 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062219    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062225    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062232    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062237    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062242    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062246    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062252    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062257    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062262    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062266    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062271    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062275    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062288    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062293    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062298    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062302    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062306    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062311    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062315    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.076446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062319    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062342    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062347    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062351    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062354    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062358    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062362    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062365    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062369    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062373    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062376    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062380    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062384    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062388    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062391    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062395    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062398    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062402    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062405    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.076915 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062409    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062413    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062417    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062420    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062424    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062429    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062433    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062444    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062449    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062453    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062457    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062461    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062465    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062469    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062473    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062476    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062481    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062485    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062490    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.077512 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062494    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062498    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062502    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062506    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062510    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062515    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062519    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062523    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.062565    5244 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062874    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062884    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062889    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062893    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062898    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062902    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062906    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.078119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062910    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062914    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062919    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062923    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062927    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062939    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062944    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062948    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062952    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062957    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062963    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062968    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062973    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062977    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062982    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062986    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062990    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062995    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.062999    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.078817 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063003    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063007    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063011    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063015    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063020    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063024    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063028    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063032    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063043    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063048    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063052    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063056    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063060    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063065    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063069    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063073    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063077    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063081    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063085    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063098    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.079312 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063103    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063107    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063112    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063116    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063120    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063127    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063131    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063134    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063139    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063143    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063147    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063151    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063156    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063160    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063164    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063168    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063172    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063176    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063181    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063185    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.079822 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063189    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063193    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063198    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063201    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063206    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063210    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063214    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063218    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063222    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063226    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063231    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063235    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063246    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063251    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063255    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063259    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063263    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063269    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063273    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063277    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.080298 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063282    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063286    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063290    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063294    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063298    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063302    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063307    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063316    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063321    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063344    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063349    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063354    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063358    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063362    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063366    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063370    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063374    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063379    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063385    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.080799 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063391    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.063433    5244 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063498    5244 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063504    5244 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063509    5244 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063514    5244 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063526    5244 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063532    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063536    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063541    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063545    5244 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063551    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063555    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063559    5244 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063563    5244 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063567    5244 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 21:15:29.081265 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063571    5244 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063576    5244 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063580    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063584    5244 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063590    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063594    5244 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063598    5244 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063602    5244 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063606    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063610    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063614    5244 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063618    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063622    5244 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063626    5244 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063631    5244 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063635    5244 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063639    5244 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063643    5244 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063648    5244 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063652    5244 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 21:15:29.081660 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063656    5244 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063660    5244 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063664    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063676    5244 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063680    5244 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063684    5244 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063688    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063694    5244 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063698    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063701    5244 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063706    5244 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063710    5244 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063714    5244 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063719    5244 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063723    5244 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063727    5244 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063732    5244 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063736    5244 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063740    5244 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063744    5244 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 21:15:29.082182 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063748    5244 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063753    5244 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063757    5244 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063761    5244 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063765    5244 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063769    5244 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063773    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063778    5244 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063782    5244 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063786    5244 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063790    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063795    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063800    5244 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063804    5244 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063808    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063813    5244 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063825    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063829    5244 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063833    5244 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063840    5244 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 21:15:29.082750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063846    5244 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063852    5244 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063857    5244 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063862    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063866    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063870    5244 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063874    5244 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063878    5244 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063884    5244 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063888    5244 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063892    5244 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063896    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063901    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063905    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063909    5244 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063914    5244 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063917    5244 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063921    5244 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063926    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 21:15:29.083227 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063932    5244 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063938    5244 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063942    5244 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063947    5244 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063951    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063955    5244 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063959    5244 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063964    5244 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063968    5244 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063972    5244 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063983    5244 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063988    5244 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:29.063992    5244 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.064097    5244 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.065732    5244 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.065816    5244 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.066093    5244 server.go:1014] "Starting client certificate rotation"
Nov 03 21:15:29.083739 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.066239    5244 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.066371    5244 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 21:09:16 +0000 UTC" deadline="2028-07-17 17:35:15.629351048 +0000 UTC"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.066437    5244 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14924h19m46.562916069s"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.066991    5244 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.069819    5244 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.071181    5244 log.go:25] "Validated CRI v1 runtime API"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.073213    5244 log.go:25] "Validated CRI v1 image API"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.074072    5244 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 21:15:29.084176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.076063    5244 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 919f8396-d4f3-4bca-a0d9-06d1eca19f80:/dev/nvme0n1p3 fd84755a-d72a-45e9-b697-9ca167a9f13c:/dev/nvme0n1p4]
Nov 03 21:15:29.084514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.076088    5244 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cfa1bf55155702b4e2b5e97acb4c607054f8fb029d8863e2099b9b92aec39a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cfa1bf55155702b4e2b5e97acb4c607054f8fb029d8863e2099b9b92aec39a0/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01063714-6865-449c-9726-caccbd2bfcc8/volumes/kubernetes.io~projected/kube-api-access-m5dk7:{mountpoint:/var/lib/kubelet/pods/01063714-6865-449c-9726-caccbd2bfcc8/volumes/kubernetes.io~projected/kube-api-access-m5dk7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01063714-6865-449c-9726-caccbd2bfcc8/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/01063714-6865-449c-9726-caccbd2bfcc8/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d1e36e3-40f2-4c43-a009-e323301c57e6/volumes/kubernetes.io~projected/kube-api-access-ptdl9:{mountpoint:/var/lib/kubelet/pods/0d1e36e3-40f2-4c43-a009-e323301c57e6/volumes/kubernetes.io~projected/kube-api-access-ptdl9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1424c53b-1fc3-4c31-841d-f2db4741152a/volumes/kubernetes.io~projected/kube-api-access-fg44m:{mountpoint:/var/lib/kubelet/pods/1424c53b-1fc3-4c31-841d-f2db4741152a/volumes/kubernetes.io~projected/kube-api-access-fg44m major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1424c53b-1fc3-4c31-841d-f2db4741152a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1424c53b-1fc3-4c31-841d-f2db4741152a/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~projected/kube-api-access-fzlp9:{mountpoint:/var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~projected/kube-api-access-fzlp9 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3408dd67-d555-4659-b48e-2366768b05c8/volumes/kubernetes.io~projected/kube-api-access-skx9b:{mountpoint:/var/lib/kubelet/pods/3408dd67-d555-4659-b48e-2366768b05c8/volumes/kubernetes.io~projected/kube-api-access-skx9b major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56dd4c25-1333-40f6-867d-1f02ff44b0d9/volumes/kubernetes.io~projected/kube-api-access-pkzbq:{mountpoint:/var/lib/kubelet/pods/56dd4c25-1333-40f6-867d-1f02ff44b0d9/volumes/kubernetes.io~projected/kube-api-access-pkzbq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5718e9a0-0924-41fe-a482-557054019ca9/volumes/kubernetes.io~projected/kube-api-access-lksds:{mountpoint:/var/lib/kubelet/pods/5718e9a0-0924-41fe-a482-557054019ca9/volumes/kubernetes.io~projected/kube-api-access-lksds major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5eeb32df-a293-4351-8743-a3b12b67dd87/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5eeb32df-a293-4351-8743-a3b12b67dd87/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62d2b800-0c2b-49ca-906e-ec29326e1726/volumes/kubernetes.io~projected/kube-api-access-5wdtj:{mountpoint:/var/lib/kubelet/pods/62d2b800-0c2b-49ca-906e-ec29326e1726/volumes/kubernetes.io~projected/kube-api-access-5wdtj major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62d2b800-0c2b-49ca-906e-ec29326e1726/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/62d2b800-0c2b-49ca-906e-ec29326e1726/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~projected/kube-api-access-qc76r:{mountpoint:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~projected/kube-api-access-qc76r major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~projected/kube-api-access-7jw65:{mountpoint:/var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~projected/kube-api-access-7jw65 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8541995b-77c8-4973-9736-f750edf753cb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8541995b-77c8-4973-9736-f750edf753cb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa443bcf-d427-46f2-b924-beb20bd4df7b/volumes/kubernetes.io~projected/kube-api-access-5xkmh:{mountpoint:/var/lib/kubelet/pods/aa443bcf-d427-46f2-b924-beb20bd4df7b/volumes/kubernetes.io~projected/kube-api-access-5xkmh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volumes/kubernetes.io~projected/kube-api-access-5vkhz:{mountpoint:/var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volumes/kubernetes.io~projected/kube-api-access-5vkhz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be40fcfd-b776-4ede-ad34-e4be87a007e0/volumes/kubernetes.io~projected/kube-api-access-lkrdx:{mountpoint:/var/lib/kubelet/pods/be40fcfd-b776-4ede-ad34-e4be87a007e0/volumes/kubernetes.io~projected/kube-api-access-lkrdx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be40fcfd-b776-4ede-ad34-e4be87a007e0/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/be40fcfd-b776-4ede-ad34-e4be87a007e0/volumes/kubernetes.io~secret/metrics-certs major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~projected/kube-api-access-d4mtm:{mountpoint:/var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~projected/kube-api-access-d4mtm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8bfe6b0-1d4f-47db-a462-a1a676845dcf/volumes/kubernetes.io~projected/kube-api-access-9b8wf:{mountpoint:/var/lib/kubelet/pods/d8bfe6b0-1d4f-47db-a462-a1a676845dcf/volumes/kubernetes.io~projected/kube-api-access-9b8wf major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~projected/kube-api-access-fgtsb:{mountpoint:/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~projected/kube-api-access-fgtsb major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7533013-9173-474b-9baf-1da837310ea1/volumes/kubernetes.io~projected/kube-api-access-dgrj7:{mountpoint:/var/lib/kubelet/pods/e7533013-9173-474b-9baf-1da837310ea1/volumes/kubernetes.io~projected/kube-api-access-dgrj7 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7533013-9173-474b-9baf-1da837310ea1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e7533013-9173-474b-9baf-1da837310ea1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb673cf5-6137-4a72-8910-e957e3685a71/volumes/kubernetes.io~projected/kube-api-access-tjn8n:{mountpoint:/var/lib/kubelet/pods/eb673cf5-6137-4a72-8910-e957e3685a71/volumes/kubernetes.io~projected/kube-api-access-tjn8n major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1470d2f7e5946718ed6bd14e7cc63f5c238fec648a8150201b83f25934e49295/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/426efd29e66534d4594d7b35803e62940108798bcda8598f2a821585c68b5f5a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f30cd618e749d3baf8ffb7bcd35ec921f4d3923d144a357f6ba348cff886d82a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/bc1754ab97374ac42deb9db80a2d96345ba3d9340810836878c83757c883ad89/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/1fe145fa7ca7dcfa828fc42b7a4190deabbfde6d32f26eae3c78195ac93624dc/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c183de2b4cd3af2dfa1a1d829a2e89fac223aa1e0320bbe0bc211e3c036db275/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/03476e8abcf77d787051b9d7f9d5df9b1709fb907658c73f37d85c8420f759ed/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fbf2350c94acf96c38acebcabeb595a168c12690c3987ba46af6195cf7bf87b6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cf0abbb43b70cd30a7f3480f3fdbb426bd5a92526387f4c658b57c8805bdade5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/5a4d5c28dacf402a73b0bd3d7404e472e3b44eb50764d974dea04caea3b208f6/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/4c15d36b96603dbd77d64abb20b16ee048c814cf54669ab70e9eac04ccfd94f7/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/9de32b5cdad5e96e725222811528afc851fcfeda413590f01a0f4196ac32e327/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f5f4b0b5fde0c1c2bdf59796e82a260032b1ac1a5e6bec52ae555244610b4612/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/40a6947cc322a1fb7050837a0e8d7eea6203fa9a74b1918cc16e887edff5dede/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/535ef04496bca1bd9dd8a0684f97a61ea953b4679f3f12c7745bb69c171826c9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a499839f43899690070ffcc1f4984de3d6f1bc0c60372e059e1f82d829fcfe66/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/afce57ed2e367314046fb766af11868142a4858945a24aa5b197661a31900e2b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/75de6c5bf2e6a1967c5443ec682a7fe7dbac8478020436b93c221055630d3b40/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2e590a28d57d415dc38909d4660b5f77bcc86f5ec2d94d373ff94c7fb398c1b9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3fdabf113e8d32eccc58f79185021eb8b491fd2c695b9522a2679b7afa491b05/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/807efb5e9133e431eb3bd040372e0e5c3980f40bbb6b2a8fa648cc73c96e3f62/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b5cd3fd4b2ce790da764b4b5ccd8a81432a35cfe26f2b2448b5719f35d17fca8/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/181be34a1a4f40a364f9733ae702a7d90594d570f516c7751ff63c86f66a53ee/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e861d5dd6edb2adbc3c23997f8514f1a5c4cad85522b8a53425696369ae74c16/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/30893e3a73272987fd543c97b8fd6ab5388348a3f1f794bccde665297764ed14/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/90b03fa11324976060292eacc0a67dcd25938e0d5dee7761de0b19276d8350e1/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/bb3855c606a9ea6c46d316842e4f380882994db4585e21055c8b7ed1b00bcade/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/de50e20376c10e98b61292c969776a148dda5d5868c448494b6f3eae5ac938b1/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/9dda5c081ff891f1da7c6c2d2f6165ca470df8a9b908cf45382f039a2546966d/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/857659d6fe74cff1df88219fea89b63e670380f39b3197a3245bc3c67f07f660/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d67a38661d35700b9b43d544c92ffdc5cf6491f703a6aaf1a4bba146c0f8c518/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f0fceaaad275d0da1fc0e932d6d9bb9f2c60d20b5e764c2ab30cd1b29e1ab9ce/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/b4c51ee6357356c9dc581254afa40212307914874a00828a2092ff10b348d8f4/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/f60e21683947d13c08a9c2d81523c11880a3c08c57ea4ef67be924801853aaea/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/534ae9ccea7e4037d02d6b13eed3a72b8b0cb05b55432e2021260f36a94f1412/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/54849165ea4590aedcf5385c6e801d62d3c69aa735b01007d40d9f5c7e146081/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/491233e20cf0f0aa71c8ed37c148d2ce258819fc62102ebb0683eabf2b5297f5/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/993b91a241a3537c23b4cca5d93551e5615336286551614fe27dcf9f8c32533d/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/69425d25712d4d9a39c0899076017532bfa787ee36775bc4c68fe4cc4f76025b/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/edbd5a78f02d10338976195a08788eb09fe1d1ec29217502149c5d12cdbc8c41/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/fce650ee50329301e85401c6c3521fa3c19a66797e632bdfaea9d381c19bc4f3/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/8c57231ca1b2a735a1ae0c60c060b2b12dbff9a7cbfd5b4af5d9d28d0ed11ff8/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/4edc8c7bf2f417001e17dcaf0633e58a221fc3de3c8a40ff0b902b9d200fc4b5/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/eb10a082033b218e1ba136ef7362f27dfede260d01e48896df82e84bd9ff0013/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/f2ae0ffa9e52c1dfc9cc2475750449010142ea57d6c10430f7a88dd27c8ee45e/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/7c85c97055653b2931e7674d63beb08bedfb4efec7bc286ad08c8d94532a9d82/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/df43a4951b7321f84a7f0f32c10ef7658b23f41fad84b188df54b7f4bf86eaab/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/70acd2424868f04c33efd2d1382fff6601799a752edcf5d820ab56f96d4f908d/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/2694c3ff7ea52398a507d91371e89406ca305e4ac98ef6a237e85f1a8d99f015/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/8a59a26019ec9664a4376f94863bfea398ed5901504bc89bbd337f721dc8531c/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/4d4e841181c288b8287d3ff576ec2f643e6aaf284ae4eb9a555aa5541e63eb90/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f597164ed2aa4d14b2a4017992c39cc7bfe136627d1ac519760f894cf3340704/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2ea056811ef576bf6d2b63f84645bcfe7fc45526aac60602fa7460d5f7682fc6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4c3ce05b24d77e08b082f14fb96f556b91be8d54124dbd628313547b35d3c858/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/02d653edf1a5de76a5e076ab3d952a9992532ac823ec17563bfe02d17d18624f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3a101c9c6d67560482315881c0798077ca8fc09fc15c86322b6974fd8931485e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f508ca730166c12f318692a255b5108b16004fd25bc4bf0095b4b9612956ccea/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f6b9d2f55b66a8534d545ecd8650a85994342e81fe8baef5b89329d32efa0f14/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4dcb233f6727dfce4f4ffcd5165d6e4fe45006982727411ca3f375188d727ff2/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/b31278d9446a310acfb56ad1b2094c47a5ed1509c743c5e2b8a53c67b2343427/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cd43f41d8bd7f779817e8e1409e30a1bdd21381c4947fc0aa673384dea2ff704/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.082998    5244 manager.go:217] Machine: {Timestamp:2026-11-03 21:15:29.082535605 +0000 UTC m=+0.066532154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098802 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec221a41e85295bfb5c1a6bcdf991939 SystemUUID:ec221a41-e852-95bf-b5c1-a6bcdf991939 BootID:6bc352d1-18e1-47e5-a6ac-595058c0f924 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/1424c53b-1fc3-4c31-841d-f2db4741152a/volumes/kubernetes.io~projected/kube-api-access-fg44m DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f96f6bddc4bd20bb684e18b36c36f101edf3e77eb0f25d989e343834b446f890/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcb0e44a9adc752439df89df51d9c122289d0ee849742a8ae7a5c53dcf1e40ec/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~projected/kube-api-access-7jw65 DeviceMajor:0 DeviceMinor:230 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~projected/kube-api-access-fgtsb DeviceMajor:0 DeviceMinor:289 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:297 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~projected/kube-api-access-qc76r DeviceMajor:0 DeviceMinor:298 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~projected/kube-api-access-d4mtm DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38579f6e734b2e0002ad7f13830e0c3d6677d9908ad5cb9dfb3db3b831461c55/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:287 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b6ba128923e70e59abedc56f6f01c24635ed5224d8753dd362c81a4ce880557/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/765a3cae7627da415f64121f31a89572494193203f70848f1a149f67c53cc504/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8bfe6b0-1d4f-47db-a462-a1a676845dcf/volumes/kubernetes.io~projected/kube-api-access-9b8wf DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/aa443bcf-d427-46f2-b924-beb20bd4df7b/volumes/kubernetes.io~projected/kube-api-access-5xkmh DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a811b19d098ddb4fb47cf31b3b5b6df9cfdbafd9a2fd178f90fb6ce20da65c0/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8541995b-77c8-4973-9736-f750edf753cb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:305 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01063714-6865-449c-9726-caccbd2bfcc8/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33a86237542c064236f266b98044cd8f4b7ed1cd86e1c60848a48cda0a6564ef/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:288 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/01063714-6865-449c-9726-caccbd2bfcc8/volumes/kubernetes.io~projected/kube-api-access-m5dk7 DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e7533013-9173-474b-9baf-1da837310ea1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:294 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05b0ec6f1d5968e637c951c5c4dc39408fc68363c1039abb267ee499d5d9a831/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d3648276-55d9-46b7-ac52-ea7223dd5868/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volumes/kubernetes.io~projected/kube-api-access-5vkhz DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:296 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90ddfaeb5d692c24a984408029c8f683837c00f52744a6bf663050a027f11dec/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e9ce98defc7689000f3c52a662ae51adb49b32182028aaf890458375a98a9a8/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56dd4c25-1333-40f6-867d-1f02ff44b0d9/volumes/kubernetes.io~projected/kube-api-access-pkzbq DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1424c53b-1fc3-4c31-841d-f2db4741152a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb673cf5-6137-4a72-8910-e957e3685a71/volumes/kubernetes.io~projected/kube-api-access-tjn8n DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6679db18-7f2c-4478-9eb1-0a20b16040e4/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:295 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/be40fcfd-b776-4ede-ad34-e4be87a007e0/volumes/kubernetes.io~projected/kube-api-access-lkrdx DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7533013-9173-474b-9baf-1da837310ea1/volumes/kubernetes.io~projected/kube-api-access-dgrj7 DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/67b25889-3fc6-4a28-99fa-267620698905/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/274de09e-a693-46cf-b317-a0cbdfcf79df/volumes/kubernetes.io~projected/kube-api-access-fzlp9 DeviceMajor:0 DeviceMinor:270 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/62d2b800-0c2b-49ca-906e-ec29326e1726/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:286 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48e805f72082851813a1ebc88a6d0938e0cd631991e566a2746eebcaa37fd136/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10defd00b50fcfe915315f945a62e588e5da96226006278ddee1b9e1f06ef413/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df08b6c224db9cd18a167d3ec525f7c50a9c519368b869200c75a5a5311abac3/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cfa1bf55155702b4e2b5e97acb4c607054f8fb029d8863e2099b9b92aec39a0/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5eeb32df-a293-4351-8743-a3b12b67dd87/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be40fcfd-b776-4ede-ad34-e4be87a007e0/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:306 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0d1e36e3-40f2-4c43-a009-e323301c57e6/volumes/kubernetes.io~projected/kube-api-access-ptdl9 DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6495fc4163839aa0ba5b6965771202a61e2f0484f0cd49561e3efc266be9aac2/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a1f125390decb030c51f01c7c0ed8a94d436a5c7fdea7e4aea75726faf0f2a3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3408dd67-d555-4659-b48e-2366768b05c8/volumes/kubernetes.io~projected/kube-api-access-skx9b DeviceMajor:0 DeviceMinor:307 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae80c711-2bb7-43e4-9bf2-521dbc9e4291/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5718e9a0-0924-41fe-a482-557054019ca9/volumes/kubernetes.io~projected/kube-api-access-lksds DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e91160f36661c28abe11f9d90f4fee16112e7696cb4deaba87ccbd77320e55c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc751f04eb449f83f463fe83e9bcebe6495ef6bd269f60ce79d2c73ddef4538b/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62d2b800-0c2b-49ca-906e-ec29326e1726/volumes/kubernetes.io~projected/kube-api-access-5wdtj DeviceMajor:0 DeviceMinor:282 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a811b19d098ddb MacAddress:ca:33:8c:8f:5a:ad Speed:10000 Mtu:8901} {Name:10defd00b50fcfe MacAddress:12:bd:54:e6:2b:a0 Speed:10000 Mtu:8901} {Name:33a86237542c064 MacAddress:46:d8:8c:ed:a9:f2 Speed:10000 Mtu:8901} {Name:38579f6e734b2e0 MacAddress:ba:f0:d2:b1:6c:81 Speed:10000 Mtu:8901} {Name:48e805f72082851 MacAddress:de:f1:7c:89:61:9e Speed:10000 Mtu:8901} {Name:4cd6bdbccbf8208 MacAddress:4a:fe:b0:f4:fc:c1 Speed:10000 Mtu:8901} {Name:6495fc4163839aa MacAddress:be:4d:b4:e6:90:c2 Speed:10000 Mtu:8901} {Name:6b6ba128923e70e MacAddress:fe:1a:77:f6:2e:21 Speed:10000 Mtu:8901} {Name:6cfa1bf55155702 MacAddress:9e:32:18:57:cf:21 Speed:10000 Mtu:8901} {Name:765a3cae7627da4 MacAddress:ea:95:70:8e:8c:21 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0d:67:d9:64:e1 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:9d:ee:e0:e1:1c Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:0d:67:d9:64:e1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:22:ec:8d:21:c0:73 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:21:f9:47:97:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083346    5244 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083398    5244 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083668    5244 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083691    5244 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-65.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083817    5244 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083824    5244 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083835    5244 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083848    5244 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.083940    5244 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084095    5244 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084195    5244 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084231    5244 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084242    5244 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084252    5244 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084261    5244 kubelet.go:405] "Adding apiserver pod source"
Nov 03 21:15:29.084847 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084275    5244 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 21:15:29.085790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084884    5244 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 21:15:29.085790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.084893    5244 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:15:29.085790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.085415    5244 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 21:15:29.089176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089155    5244 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 21:15:29.089176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089174    5244 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 21:15:29.089370 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089298    5244 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 21:15:29.089568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089555    5244 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 21:15:29.089669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089586    5244 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089676    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089689    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089695    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089701    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089706    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089711    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089717    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 21:15:29.089722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089723    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 21:15:29.090137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089730    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 21:15:29.090137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089736    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 21:15:29.090137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089745    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 21:15:29.090137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089755    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 21:15:29.090137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089778    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 21:15:29.090137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.089784    5244 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 21:15:29.093283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.093262    5244 server.go:1263] "Started kubelet"
Nov 03 21:15:29.093424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.093365    5244 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 21:15:29.093424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.093381    5244 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 21:15:29.093515 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.093457    5244 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 21:15:29.094024 ip-10-0-129-65 systemd[1]: Started Kubernetes Kubelet.
Nov 03 21:15:29.094250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.094226    5244 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 21:15:29.094610 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.094597    5244 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096439    5244 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096525    5244 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096579    5244 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 21:09:18 +0000 UTC" deadline="2028-05-26 21:43:52.72908852 +0000 UTC"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096612    5244 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13680h28m23.632479994s"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096782    5244 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096791    5244 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.096845    5244 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.096895    5244 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.097776    5244 factory.go:55] Registering systemd factory
Nov 03 21:15:29.099031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.097801    5244 factory.go:223] Registration of the systemd container factory successfully
Nov 03 21:15:29.100171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.100070    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56dd4c25-1333-40f6-867d-1f02ff44b0d9" volumeName="kubernetes.io/projected/56dd4c25-1333-40f6-867d-1f02ff44b0d9-kube-api-access-pkzbq" seLinuxMountContext=""
Nov 03 21:15:29.104597 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.104580    5244 factory.go:153] Registering CRI-O factory
Nov 03 21:15:29.104597 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.104598    5244 factory.go:223] Registration of the crio container factory successfully
Nov 03 21:15:29.104728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.104653    5244 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 21:15:29.104728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.104692    5244 factory.go:103] Registering Raw factory
Nov 03 21:15:29.104728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.104709    5244 manager.go:1199] Started watching for new ooms in manager
Nov 03 21:15:29.105143 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.105128    5244 manager.go:319] Starting recovery of all containers
Nov 03 21:15:29.106055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.100124    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62d2b800-0c2b-49ca-906e-ec29326e1726" volumeName="kubernetes.io/projected/62d2b800-0c2b-49ca-906e-ec29326e1726-kube-api-access-5wdtj" seLinuxMountContext=""
Nov 03 21:15:29.106132 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106086    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-federate-client-tls" seLinuxMountContext=""
Nov 03 21:15:29.106132 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106109    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa443bcf-d427-46f2-b924-beb20bd4df7b" volumeName="kubernetes.io/projected/aa443bcf-d427-46f2-b924-beb20bd4df7b-kube-api-access-5xkmh" seLinuxMountContext=""
Nov 03 21:15:29.106132 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106127    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274de09e-a693-46cf-b317-a0cbdfcf79df" volumeName="kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106140    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1424c53b-1fc3-4c31-841d-f2db4741152a" volumeName="kubernetes.io/secret/1424c53b-1fc3-4c31-841d-f2db4741152a-cert" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106156    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae80c711-2bb7-43e4-9bf2-521dbc9e4291" volumeName="kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106167    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3648276-55d9-46b7-ac52-ea7223dd5868" volumeName="kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-tmp" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106183    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3648276-55d9-46b7-ac52-ea7223dd5868" volumeName="kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-tuned" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106198    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106209    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8541995b-77c8-4973-9736-f750edf753cb" volumeName="kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106225    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106236    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-metrics-client-ca" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106247    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eeb32df-a293-4351-8743-a3b12b67dd87" volumeName="kubernetes.io/configmap/5eeb32df-a293-4351-8743-a3b12b67dd87-konnectivity-ca" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106271    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063714-6865-449c-9726-caccbd2bfcc8" volumeName="kubernetes.io/secret/01063714-6865-449c-9726-caccbd2bfcc8-metrics-tls" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106287    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-client-tls" seLinuxMountContext=""
Nov 03 21:15:29.106296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106298    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063714-6865-449c-9726-caccbd2bfcc8" volumeName="kubernetes.io/projected/01063714-6865-449c-9726-caccbd2bfcc8-kube-api-access-m5dk7" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106310    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5718e9a0-0924-41fe-a482-557054019ca9" volumeName="kubernetes.io/projected/5718e9a0-0924-41fe-a482-557054019ca9-kube-api-access-lksds" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106343    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae80c711-2bb7-43e4-9bf2-521dbc9e4291" volumeName="kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-config" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106355    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfe6b0-1d4f-47db-a462-a1a676845dcf" volumeName="kubernetes.io/projected/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-kube-api-access-9b8wf" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106370    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5718e9a0-0924-41fe-a482-557054019ca9" volumeName="kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106383    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b25889-3fc6-4a28-99fa-267620698905" volumeName="kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106393    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be40fcfd-b776-4ede-ad34-e4be87a007e0" volumeName="kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106409    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106422    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7533013-9173-474b-9baf-1da837310ea1" volumeName="kubernetes.io/configmap/e7533013-9173-474b-9baf-1da837310ea1-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106438    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb673cf5-6137-4a72-8910-e957e3685a71" volumeName="kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-multus-daemon-config" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106451    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1424c53b-1fc3-4c31-841d-f2db4741152a" volumeName="kubernetes.io/projected/1424c53b-1fc3-4c31-841d-f2db4741152a-kube-api-access-fg44m" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106462    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274de09e-a693-46cf-b317-a0cbdfcf79df" volumeName="kubernetes.io/empty-dir/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-textfile" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106490    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62d2b800-0c2b-49ca-906e-ec29326e1726" volumeName="kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106505    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274de09e-a693-46cf-b317-a0cbdfcf79df" volumeName="kubernetes.io/projected/274de09e-a693-46cf-b317-a0cbdfcf79df-kube-api-access-fzlp9" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106516    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b25889-3fc6-4a28-99fa-267620698905" volumeName="kubernetes.io/projected/67b25889-3fc6-4a28-99fa-267620698905-kube-api-access-7jw65" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106527    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b25889-3fc6-4a28-99fa-267620698905" volumeName="kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106544    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106556    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/projected/6679db18-7f2c-4478-9eb1-0a20b16040e4-kube-api-access-qc76r" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106575    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae80c711-2bb7-43e4-9bf2-521dbc9e4291" volumeName="kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-env-overrides" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106586    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae80c711-2bb7-43e4-9bf2-521dbc9e4291" volumeName="kubernetes.io/secret/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106600    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3648276-55d9-46b7-ac52-ea7223dd5868" volumeName="kubernetes.io/projected/d3648276-55d9-46b7-ac52-ea7223dd5868-kube-api-access-d4mtm" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106617    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb673cf5-6137-4a72-8910-e957e3685a71" volumeName="kubernetes.io/projected/eb673cf5-6137-4a72-8910-e957e3685a71-kube-api-access-tjn8n" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106630    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063714-6865-449c-9726-caccbd2bfcc8" volumeName="kubernetes.io/empty-dir/01063714-6865-449c-9726-caccbd2bfcc8-tmp-dir" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106645    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1e36e3-40f2-4c43-a009-e323301c57e6" volumeName="kubernetes.io/projected/0d1e36e3-40f2-4c43-a009-e323301c57e6-kube-api-access-ptdl9" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106656    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5eeb32df-a293-4351-8743-a3b12b67dd87" volumeName="kubernetes.io/secret/5eeb32df-a293-4351-8743-a3b12b67dd87-agent-certs" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106667    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7533013-9173-474b-9baf-1da837310ea1" volumeName="kubernetes.io/secret/e7533013-9173-474b-9baf-1da837310ea1-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106682    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274de09e-a693-46cf-b317-a0cbdfcf79df" volumeName="kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-tls" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106699    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae80c711-2bb7-43e4-9bf2-521dbc9e4291" volumeName="kubernetes.io/projected/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-kube-api-access-5vkhz" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106714    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfe6b0-1d4f-47db-a462-a1a676845dcf" volumeName="kubernetes.io/configmap/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-iptables-alerter-script" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106725    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106736    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106751    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106761    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7533013-9173-474b-9baf-1da837310ea1" volumeName="kubernetes.io/empty-dir/e7533013-9173-474b-9baf-1da837310ea1-data-volume" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106774    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb673cf5-6137-4a72-8910-e957e3685a71" volumeName="kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-cni-binary-copy" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106790    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56dd4c25-1333-40f6-867d-1f02ff44b0d9" volumeName="kubernetes.io/empty-dir/56dd4c25-1333-40f6-867d-1f02ff44b0d9-tmp-dir" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106804    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106815    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7533013-9173-474b-9baf-1da837310ea1" volumeName="kubernetes.io/projected/e7533013-9173-474b-9baf-1da837310ea1-kube-api-access-dgrj7" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106827    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274de09e-a693-46cf-b317-a0cbdfcf79df" volumeName="kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-metrics-client-ca" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106842    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67b25889-3fc6-4a28-99fa-267620698905" volumeName="kubernetes.io/configmap/67b25889-3fc6-4a28-99fa-267620698905-metrics-client-ca" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106852    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106887    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" volumeName="kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106901    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3408dd67-d555-4659-b48e-2366768b05c8" volumeName="kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106912    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5718e9a0-0924-41fe-a482-557054019ca9" volumeName="kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106927    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106939    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01063714-6865-449c-9726-caccbd2bfcc8" volumeName="kubernetes.io/configmap/01063714-6865-449c-9726-caccbd2bfcc8-config-volume" seLinuxMountContext=""
Nov 03 21:15:29.106953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106980    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d1e36e3-40f2-4c43-a009-e323301c57e6" volumeName="kubernetes.io/configmap/0d1e36e3-40f2-4c43-a009-e323301c57e6-serviceca" seLinuxMountContext=""
Nov 03 21:15:29.108926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.106997    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5718e9a0-0924-41fe-a482-557054019ca9" volumeName="kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-binary-copy" seLinuxMountContext=""
Nov 03 21:15:29.108926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.107009    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6679db18-7f2c-4478-9eb1-0a20b16040e4" volumeName="kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client" seLinuxMountContext=""
Nov 03 21:15:29.108926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.107027    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be40fcfd-b776-4ede-ad34-e4be87a007e0" volumeName="kubernetes.io/projected/be40fcfd-b776-4ede-ad34-e4be87a007e0-kube-api-access-lkrdx" seLinuxMountContext=""
Nov 03 21:15:29.108926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.107062    5244 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="274de09e-a693-46cf-b317-a0cbdfcf79df" volumeName="kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 21:15:29.108926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.107080    5244 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 21:15:29.108926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.107095    5244 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 21:15:29.126676 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.126641    5244 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 21:15:29.128653 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.128612    5244 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 21:15:29.128653 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.128635    5244 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 21:15:29.128653 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.128652    5244 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 21:15:29.128894 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.128684    5244 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 21:15:29.152991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.152957    5244 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="a34f00d7a6763d6be63467c7f34595921056010afa4a8cba99071ecc089aab5d" exitCode=0
Nov 03 21:15:29.152991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.152985    5244 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="cb5ea2d703cb0beccb4b3c08ebec5e745ab0d855811c0ac41838304156bf511f" exitCode=0
Nov 03 21:15:29.152991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.152991    5244 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="e8030149cbce3636ac3b9d8706ee048032c7d0eff9302bde3158c5e86eda6357" exitCode=0
Nov 03 21:15:29.152991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.153000    5244 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="3deed723913e84bb2192875fae0e72c083e6a7ed552e964a3783d438fab7a384" exitCode=0
Nov 03 21:15:29.153389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.153006    5244 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="2cc26764ce0b508e27e20775151cbb1af7e771a1f9c72da6f0989dda8fac3ca7" exitCode=0
Nov 03 21:15:29.153389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.153012    5244 generic.go:356] "Generic (PLEG): container finished" podID="5718e9a0-0924-41fe-a482-557054019ca9" containerID="d6027ce2220e8a691d54b0374bc94b2a3569bddfaa029b5a63862e99d6ad43be" exitCode=0
Nov 03 21:15:29.157912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.157880    5244 generic.go:356] "Generic (PLEG): container finished" podID="f2f624a3552b5bcbd4b3914b37486b5a" containerID="476dcf37c26782319d2e02475ca3c536352ac5b86b9f5f3b7bcba4c42376fd20" exitCode=0
Nov 03 21:15:29.164508 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.164476    5244 generic.go:356] "Generic (PLEG): container finished" podID="274de09e-a693-46cf-b317-a0cbdfcf79df" containerID="12fa484184466184676e48761907402b24f8fa45cc853ef85d9c38fa54a7540d" exitCode=0
Nov 03 21:15:29.196932 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.196906    5244 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:15:29.201777 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.201749    5244 manager.go:324] Recovery completed
Nov 03 21:15:29.219919 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.219379    5244 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:15:29.221294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.221271    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:15:29.221412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.221304    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:15:29.221412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.221315    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:15:29.224047 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224029    5244 cpu_manager.go:226] "Starting" policy="none"
Nov 03 21:15:29.224047 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224045    5244 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 21:15:29.224158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224049    5244 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:15:29.224158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224068    5244 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 21:15:29.224256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224227    5244 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 21:15:29.224289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224246    5244 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 21:15:29.224289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224264    5244 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 21:15:29.224289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224270    5244 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 21:15:29.224289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224275    5244 policy_none.go:50] "Start"
Nov 03 21:15:29.224289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224284    5244 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 21:15:29.224452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224294    5244 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 21:15:29.224488 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224478    5244 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 21:15:29.224517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224490    5244 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 21:15:29.224517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.224496    5244 policy_none.go:44] "Start"
Nov 03 21:15:29.225045 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.225026    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:15:29.225088 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.225075    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:15:29.225122 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.225095    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:15:29.228784 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.228762    5244 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 21:15:29.229439 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.229417    5244 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 21:15:29.232568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.232552    5244 manager.go:341] "Starting Device Plugin manager"
Nov 03 21:15:29.232649 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.232581    5244 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 21:15:29.232649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.232593    5244 server.go:87] "Starting device plugin registration server"
Nov 03 21:15:29.232888 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.232874    5244 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 21:15:29.232963 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.232888    5244 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 21:15:29.233012 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.232980    5244 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 21:15:29.233998 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.233107    5244 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 21:15:29.234151 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.234010    5244 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 21:15:29.234151 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.234019    5244 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 21:15:29.234151 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.234080    5244 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T21:15:29.233990778Z","UUID":"a196d9b7-8196-485e-a7ef-106e54bb6e14","Handler":null,"Name":"","Endpoint":""}
Nov 03 21:15:29.234999 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.234960    5244 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 21:15:29.235072 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.235021    5244 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-65.us-west-1.compute.internal\" not found"
Nov 03 21:15:29.235304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.235288    5244 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 21:15:29.235408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.235316    5244 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 21:15:29.290800 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.290773    5244 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 21:15:29.292978 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.292933    5244 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:15:29.292978 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.292931    5244 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:15:29.333609 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.333528    5244 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 21:15:29.334697 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.334674    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 21:15:29.334817 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.334715    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 21:15:29.334817 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.334730    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 21:15:29.334817 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.334782    5244 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.340377 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.340357    5244 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:15:29.390240 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.390207    5244 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 21:15:29.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429140    5244 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"]
Nov 03 21:15:29.429865 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429808    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"f2f624a3552b5bcbd4b3914b37486b5a","Type":"ContainerStarted","Data":"7ea07e6f822a5b939db961b17be5d6f8e3cdd8c7cd0b08ee2506c9be0c92de99"}
Nov 03 21:15:29.429946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429889    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"f2f624a3552b5bcbd4b3914b37486b5a","Type":"ContainerDied","Data":"476dcf37c26782319d2e02475ca3c536352ac5b86b9f5f3b7bcba4c42376fd20"}
Nov 03 21:15:29.429946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429915    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"f2f624a3552b5bcbd4b3914b37486b5a","Type":"ContainerStarted","Data":"7cc10fd440c50e5d5e7afe5cdef3aa7833e01fb40f41f2b0c3fdb66956219efb"}
Nov 03 21:15:29.430025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429952    5244 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.430025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429968    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"29b854c94d02e7fd862ea1754302117f","Type":"ContainerStarted","Data":"cf66ecd86aa43b38ec7e23e6745d4a78640b6196646f72280845862537d00cbb"}
Nov 03 21:15:29.430025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.429989    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal" event={"ID":"29b854c94d02e7fd862ea1754302117f","Type":"ContainerStarted","Data":"1fa18a64c1268e21e6d0b1f7c60e147a5293f45e702cf2bfc4f041ceb6e7b7a2"}
Nov 03 21:15:29.430126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.430064    5244 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.447132 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.447109    5244 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.447250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.447183    5244 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.447842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.447823    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeNotReady"
Nov 03 21:15:29.447897 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.447857    5244 setters.go:546] "Node became not ready" node="ip-10-0-129-65.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T21:15:29Z","lastTransitionTime":"2026-11-03T21:15:29Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 21:15:29.489014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.488978    5244 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:15:29.489172 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.489043    5244 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.489476 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.489465    5244 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:15:29.489510 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:29.489488    5244 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.502637 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.502605    5244 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-65.us-west-1.compute.internal" is owned by different node (oldNodeID="6dd1388c-d469-431d-981b-7f58b1782789", newNodeID=""), cleaning up...
Nov 03 21:15:29.592131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.592051    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.592131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.592088    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.592131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.592110    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29b854c94d02e7fd862ea1754302117f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"29b854c94d02e7fd862ea1754302117f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.692255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.692223    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.692255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.692255    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.692472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.692277    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29b854c94d02e7fd862ea1754302117f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"29b854c94d02e7fd862ea1754302117f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.692472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.692323    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29b854c94d02e7fd862ea1754302117f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"29b854c94d02e7fd862ea1754302117f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.692472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.692368    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:29.692472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:29.692368    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2f624a3552b5bcbd4b3914b37486b5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" (UID: \"f2f624a3552b5bcbd4b3914b37486b5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:30.085119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.085088    5244 apiserver.go:52] "Watching apiserver"
Nov 03 21:15:30.190007 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.189965    5244 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 21:15:30.190633 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.190603    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gg868","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm","openshift-insights/insights-runtime-extractor-cvcz7","openshift-ovn-kubernetes/ovnkube-node-gmxs8","openshift-dns/node-resolver-g7cv4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal","openshift-monitoring/node-exporter-xr2gw","openshift-network-diagnostics/network-check-target-kgjns","openshift-monitoring/telemeter-client-964cff64-qf6zt","openshift-network-operator/iptables-alerter-wrgf2","openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2","openshift-multus/multus-additional-cni-plugins-fp2r2","openshift-cluster-node-tuning-operator/tuned-zv8ps","openshift-ingress-canary/ingress-canary-fssfw","kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal","openshift-monitoring/monitoring-plugin-77f4f76594-gv74f","openshift-multus/network-metrics-daemon-tjq5q","kube-system/global-pull-secret-syncer-s77gw","openshift-dns/dns-default-6jq4g","kube-system/konnectivity-agent-h4zdr","openshift-console/console-78789694d8-xqjq4","openshift-image-registry/node-ca-pshfv"]
Nov 03 21:15:30.193281 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.193248    5244 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:30.193485 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.193467    5244 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:30.248680 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.248572    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 21:15:30.248855 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.248793    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lzq4j\""
Nov 03 21:15:30.249117 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249097    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4rzx\""
Nov 03 21:15:30.249645 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249618    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cldzt\""
Nov 03 21:15:30.249645 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249637    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 21:15:30.249814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249685    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.249814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249719    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 21:15:30.249940 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249922    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 21:15:30.250005 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249930    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.250005 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249968    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 21:15:30.250005 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.249987    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78m6\""
Nov 03 21:15:30.250140 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250120    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4f9ng\""
Nov 03 21:15:30.250251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250173    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.250251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250198    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zbt6z\""
Nov 03 21:15:30.250251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250203    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250293    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250310    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbwk2\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250375    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250376    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250395    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250380    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250423    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250431    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250441    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.250444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250450    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.250858 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250461    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 21:15:30.250858 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250489    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.250858 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.250541    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.252230 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.252212    5244 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:15:30.252297 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.252265    5244 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:30.261699 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.261675    5244 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 21:15:30.261828 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.261731    5244 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.us-west-1.compute.internal"
Nov 03 21:15:30.296988 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.296940    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-k8s-cni-cncf-io\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.297180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.296995    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-kubernetes\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.297180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297030    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw65\" (UniqueName: \"kubernetes.io/projected/67b25889-3fc6-4a28-99fa-267620698905-kube-api-access-7jw65\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.297180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297058    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1424c53b-1fc3-4c31-841d-f2db4741152a-cert\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:30.297180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297100    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-multus-certs\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.297180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297128    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-systemd\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.297180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297154    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-netd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.297547 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297261    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-cni-binary-copy\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.297547 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297378    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdl9\" (UniqueName: \"kubernetes.io/projected/0d1e36e3-40f2-4c43-a009-e323301c57e6-kube-api-access-ptdl9\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.297626 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297599    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-cni-binary-copy\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.297626 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297601    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dk7\" (UniqueName: \"kubernetes.io/projected/01063714-6865-449c-9726-caccbd2bfcc8-kube-api-access-m5dk7\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.297728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297641    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mtm\" (UniqueName: \"kubernetes.io/projected/d3648276-55d9-46b7-ac52-ea7223dd5868-kube-api-access-d4mtm\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.297728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297666    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-systemd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.297728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297692    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-iptables-alerter-script\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.297728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297709    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrj7\" (UniqueName: \"kubernetes.io/projected/e7533013-9173-474b-9baf-1da837310ea1-kube-api-access-dgrj7\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.297904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297728    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b25889-3fc6-4a28-99fa-267620698905-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.297904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297752    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-multus\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.297904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297848    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-etc-kubernetes\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.298040 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297906    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d1e36e3-40f2-4c43-a009-e323301c57e6-serviceca\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.298040 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.297998    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.298124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298083    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-sys-fs\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.298170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298129    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-etc-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.298170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298164    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-log-socket\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.298227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298204    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkhz\" (UniqueName: \"kubernetes.io/projected/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-kube-api-access-5vkhz\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.298359 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298310    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-system-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.298462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298388    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.298462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298402    5244 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 21:15:30.298462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298416    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-sys\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.298570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298461    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-tmp\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.298570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298488    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56dd4c25-1333-40f6-867d-1f02ff44b0d9-hosts-file\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.298570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298526    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eeb32df-a293-4351-8743-a3b12b67dd87-agent-certs\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:15:30.298667 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298602    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-tmp\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.298697 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298672    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01063714-6865-449c-9726-caccbd2bfcc8-config-volume\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.298726 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298700    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0d1e36e3-40f2-4c43-a009-e323301c57e6-serviceca\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.298726 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298716    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.298782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298766    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eeb32df-a293-4351-8743-a3b12b67dd87-agent-certs\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:15:30.298782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298770    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-run\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.298842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298798    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-bin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.298842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298816    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-hostroot\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.298902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298844    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.298902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298869    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.298902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298887    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.298984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298910    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:30.298984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298957    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn8n\" (UniqueName: \"kubernetes.io/projected/eb673cf5-6137-4a72-8910-e957e3685a71-kube-api-access-tjn8n\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.298984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.298975    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.299067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299003    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksds\" (UniqueName: \"kubernetes.io/projected/5718e9a0-0924-41fe-a482-557054019ca9-kube-api-access-lksds\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.299067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299028    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-conf\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.299067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299053    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-host\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.299160 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299111    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-env-overrides\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.299228 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299210    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:30.299273 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299215    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be40fcfd-b776-4ede-ad34-e4be87a007e0-metrics-certs\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:30.299273 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299247    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8wf\" (UniqueName: \"kubernetes.io/projected/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-kube-api-access-9b8wf\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.299353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299293    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-env-overrides\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.299388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299368    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-conf-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.299425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299387    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d1e36e3-40f2-4c43-a009-e323301c57e6-host\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.299425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299402    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-systemd-units\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.299495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299440    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovn-node-metrics-cert\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.299495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299464    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dd4c25-1333-40f6-867d-1f02ff44b0d9-tmp-dir\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.299495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299483    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkzbq\" (UniqueName: \"kubernetes.io/projected/56dd4c25-1333-40f6-867d-1f02ff44b0d9-kube-api-access-pkzbq\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.299609 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299590    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7533013-9173-474b-9baf-1da837310ea1-data-volume\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.299668 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299652    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-lib-modules\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.299706 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299678    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7533013-9173-474b-9baf-1da837310ea1-data-volume\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.299706 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299679    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-kubelet\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.299774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299679    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovn-node-metrics-cert\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.299774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299709    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-multus-daemon-config\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.299774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299726    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dd4c25-1333-40f6-867d-1f02ff44b0d9-tmp-dir\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.299774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299737    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-device-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.299774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299758    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkmh\" (UniqueName: \"kubernetes.io/projected/aa443bcf-d427-46f2-b924-beb20bd4df7b-kube-api-access-5xkmh\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299789    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-kubelet-config\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299825    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299843    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-netns\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299859    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299868    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb673cf5-6137-4a72-8910-e957e3685a71-multus-daemon-config\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299874    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-socket-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.299916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299898    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-cnibin\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.300141 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299947    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01063714-6865-449c-9726-caccbd2bfcc8-metrics-tls\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.300141 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.299966    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-modprobe-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.300141 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300011    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-slash\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300141 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300070    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-netns\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300141 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300090    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-var-lib-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300141 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300114    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300154    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-config\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300241    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.300368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300281    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb32df-a293-4351-8743-a3b12b67dd87-konnectivity-ca\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:15:30.300368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300308    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg44m\" (UniqueName: \"kubernetes.io/projected/1424c53b-1fc3-4c31-841d-f2db4741152a-kube-api-access-fg44m\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300370    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01063714-6865-449c-9726-caccbd2bfcc8-tmp-dir\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300384    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-config\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300404    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-ovn\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300427    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-bin\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300479    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01063714-6865-449c-9726-caccbd2bfcc8-tmp-dir\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300491    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7533013-9173-474b-9baf-1da837310ea1-crio-socket\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300512    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.300550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300547    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300597    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eeb32df-a293-4351-8743-a3b12b67dd87-konnectivity-ca\") pod \"konnectivity-agent-h4zdr\" (UID: \"5eeb32df-a293-4351-8743-a3b12b67dd87\") " pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300599    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-script-lib\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300648    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrdx\" (UniqueName: \"kubernetes.io/projected/be40fcfd-b776-4ede-ad34-e4be87a007e0-kube-api-access-lkrdx\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300678    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-cnibin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300707    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-registration-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300733    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysconfig\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300782    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-tuned\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300802    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-ovnkube-script-lib\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300807    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-kubelet\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.300861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300842    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-node-log\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300873    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-host-slash\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300894    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7533013-9173-474b-9baf-1da837310ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300895    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-tuned\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300924    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-socket-dir-parent\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.300970    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-dbus\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.301007    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.301037    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-var-lib-kubelet\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.301064    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7533013-9173-474b-9baf-1da837310ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.301090    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.301346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.301224    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.348583 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.348498    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.349011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.348986    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.349135 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.349120    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.349198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.349137    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 21:15:30.349198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.349150    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.349606 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.349592    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 21:15:30.349638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.349628    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.349883 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.349870    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 21:15:30.350533 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350516    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 21:15:30.350587 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350537    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 21:15:30.350587 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350544    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 21:15:30.350587 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350562    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 21:15:30.350682 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350621    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.350766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350749    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 21:15:30.350811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350767    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.350811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350759    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.350811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350768    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.350811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.350751    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.351050 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351035    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 21:15:30.351050 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351047    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351046    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351094    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wqd2s\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351100    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351107    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351112    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351128    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351128    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351148    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rf5\""
Nov 03 21:15:30.351172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351139    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79p29\""
Nov 03 21:15:30.351469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351228    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 21:15:30.351469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351369    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7533013-9173-474b-9baf-1da837310ea1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.351535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351522    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/67b25889-3fc6-4a28-99fa-267620698905-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.351776 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351746    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rdfhn\""
Nov 03 21:15:30.351882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351780    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m96vh\""
Nov 03 21:15:30.351882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351787    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 21:15:30.351882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.351810    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rbksq\""
Nov 03 21:15:30.352110 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352094    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzfft\""
Nov 03 21:15:30.352160 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352128    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxfhg\""
Nov 03 21:15:30.352191 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352159    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 21:15:30.352191 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352173    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 21:15:30.352252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352219    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8l7\""
Nov 03 21:15:30.352300 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352252    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 21:15:30.352300 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352285    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jnct2\""
Nov 03 21:15:30.352425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352367    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 21:15:30.352425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352377    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tq7pf\""
Nov 03 21:15:30.352425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352418    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trdz7\""
Nov 03 21:15:30.352545 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352436    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 21:15:30.352545 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.352456    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 21:15:30.358188 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.358169    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1424c53b-1fc3-4c31-841d-f2db4741152a-cert\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:30.358188 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.358179    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67b25889-3fc6-4a28-99fa-267620698905-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.358300 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.358173    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-iptables-alerter-script\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.359355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.359318    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01063714-6865-449c-9726-caccbd2bfcc8-config-volume\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.359391 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.359347    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.359474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.359459    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8541995b-77c8-4973-9736-f750edf753cb-original-pull-secret\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.360504 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.360487    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5718e9a0-0924-41fe-a482-557054019ca9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.360558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.360492    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01063714-6865-449c-9726-caccbd2bfcc8-metrics-tls\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.361647 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.361632    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7533013-9173-474b-9baf-1da837310ea1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.399094 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399060    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrj7\" (UniqueName: \"kubernetes.io/projected/e7533013-9173-474b-9baf-1da837310ea1-kube-api-access-dgrj7\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.399313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399062    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkhz\" (UniqueName: \"kubernetes.io/projected/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-kube-api-access-5vkhz\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.399313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399239    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdl9\" (UniqueName: \"kubernetes.io/projected/0d1e36e3-40f2-4c43-a009-e323301c57e6-kube-api-access-ptdl9\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.399494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399354    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dk7\" (UniqueName: \"kubernetes.io/projected/01063714-6865-449c-9726-caccbd2bfcc8-kube-api-access-m5dk7\") pod \"dns-default-6jq4g\" (UID: \"01063714-6865-449c-9726-caccbd2bfcc8\") " pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.399713 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399690    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 21:15:30.399814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399727    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mtm\" (UniqueName: \"kubernetes.io/projected/d3648276-55d9-46b7-ac52-ea7223dd5868-kube-api-access-d4mtm\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.399814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399755    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn8n\" (UniqueName: \"kubernetes.io/projected/eb673cf5-6137-4a72-8910-e957e3685a71-kube-api-access-tjn8n\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.399814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.399757    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksds\" (UniqueName: \"kubernetes.io/projected/5718e9a0-0924-41fe-a482-557054019ca9-kube-api-access-lksds\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.400083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400052    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8wf\" (UniqueName: \"kubernetes.io/projected/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-kube-api-access-9b8wf\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.400170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400082    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkzbq\" (UniqueName: \"kubernetes.io/projected/56dd4c25-1333-40f6-867d-1f02ff44b0d9-kube-api-access-pkzbq\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.400170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400058    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skx9b\" (UniqueName: \"kubernetes.io/projected/3408dd67-d555-4659-b48e-2366768b05c8-kube-api-access-skx9b\") pod \"network-check-target-kgjns\" (UID: \"3408dd67-d555-4659-b48e-2366768b05c8\") " pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:30.400280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400196    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrdx\" (UniqueName: \"kubernetes.io/projected/be40fcfd-b776-4ede-ad34-e4be87a007e0-kube-api-access-lkrdx\") pod \"network-metrics-daemon-tjq5q\" (UID: \"be40fcfd-b776-4ede-ad34-e4be87a007e0\") " pod="openshift-multus/network-metrics-daemon-tjq5q"
Nov 03 21:15:30.400280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400265    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkmh\" (UniqueName: \"kubernetes.io/projected/aa443bcf-d427-46f2-b924-beb20bd4df7b-kube-api-access-5xkmh\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.400468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400451    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg44m\" (UniqueName: \"kubernetes.io/projected/1424c53b-1fc3-4c31-841d-f2db4741152a-kube-api-access-fg44m\") pod \"ingress-canary-fssfw\" (UID: \"1424c53b-1fc3-4c31-841d-f2db4741152a\") " pod="openshift-ingress-canary/ingress-canary-fssfw"
Nov 03 21:15:30.400519 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.400465    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw65\" (UniqueName: \"kubernetes.io/projected/67b25889-3fc6-4a28-99fa-267620698905-kube-api-access-7jw65\") pod \"prometheus-operator-9cd89b74c-rdmv2\" (UID: \"67b25889-3fc6-4a28-99fa-267620698905\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rdmv2"
Nov 03 21:15:30.401317 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401302    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-conf-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.401382 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401339    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d1e36e3-40f2-4c43-a009-e323301c57e6-host\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.401382 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401358    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-kubelet\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.401382 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401375    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-kubelet-config\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.401462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401390    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-federate-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.401462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401393    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d1e36e3-40f2-4c43-a009-e323301c57e6-host\") pod \"node-ca-pshfv\" (UID: \"0d1e36e3-40f2-4c43-a009-e323301c57e6\") " pod="openshift-image-registry/node-ca-pshfv"
Nov 03 21:15:30.401462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401406    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlp9\" (UniqueName: \"kubernetes.io/projected/274de09e-a693-46cf-b317-a0cbdfcf79df-kube-api-access-fzlp9\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.401462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401424    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-kubelet\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.401462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401393    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-conf-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.401462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401442    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-netns\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401478    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401495    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-socket-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401520    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-var-lib-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401534    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401531    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-netns\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401546    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401489    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-kubelet-config\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401559    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-socket-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401570    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401570    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-var-lib-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401600    5244 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401595    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401611    5244 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401635    5244 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401660    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.901645731 +0000 UTC m=+1.885642268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-config" not registered
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401662    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-federate-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401684    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.90167332 +0000 UTC m=+1.885669858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"service-ca" not registered
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401637    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc76r\" (UniqueName: \"kubernetes.io/projected/6679db18-7f2c-4478-9eb1-0a20b16040e4-kube-api-access-qc76r\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401705    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.901691904 +0000 UTC m=+1.885688441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401728    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-bin\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.401734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401746    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-cnibin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401763    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysconfig\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401787    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-host-slash\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401805    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-bin\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401814    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-var-lib-kubelet\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401831    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-cnibin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401838    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysconfig\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401846    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8bfe6b0-1d4f-47db-a462-a1a676845dcf-host-slash\") pod \"iptables-alerter-wrgf2\" (UID: \"d8bfe6b0-1d4f-47db-a462-a1a676845dcf\") " pod="openshift-network-operator/iptables-alerter-wrgf2"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401840    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401881    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401905    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.401885    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-var-lib-kubelet\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.401958    5244 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402021    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-kubernetes\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.402081    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.902057138 +0000 UTC m=+1.886053677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402094    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-kubernetes\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402117    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-metrics-client-ca\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402114    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402159    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-multus-certs\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402191    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-netd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402230    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-etc-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402232    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-multus-certs\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402282    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-log-socket\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402322    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-metrics-client-ca\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402324    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402357    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-cni-netd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402387    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56dd4c25-1333-40f6-867d-1f02ff44b0d9-hosts-file\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402397    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-etc-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402362    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-log-socket\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402399    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402404    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402449    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402452    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56dd4c25-1333-40f6-867d-1f02ff44b0d9-hosts-file\") pod \"node-resolver-g7cv4\" (UID: \"56dd4c25-1333-40f6-867d-1f02ff44b0d9\") " pod="openshift-dns/node-resolver-g7cv4"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402477    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-hostroot\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402507    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402538    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402572    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-host\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402575    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-accelerators-collector-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402580    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402537    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-hostroot\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402613    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-systemd-units\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402624    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402635    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-host\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.402681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402651    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402727    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-systemd-units\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402765    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402804    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-lib-modules\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402836    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-device-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402858    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-modprobe-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402880    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-ovn\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402907    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-device-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402917    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7533013-9173-474b-9baf-1da837310ea1-crio-socket\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402953    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402981    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-cnibin\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403007    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-lib-modules\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403017    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-ovn\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403027    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-cnibin\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403038    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-slash\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403051    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-modprobe-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403069    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-netns\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403026    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-client-tls\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.402960    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7533013-9173-474b-9baf-1da837310ea1-crio-socket\") pod \"insights-runtime-extractor-cvcz7\" (UID: \"e7533013-9173-474b-9baf-1da837310ea1\") " pod="openshift-insights/insights-runtime-extractor-cvcz7"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403090    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-textfile\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403078    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-slash\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403126    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-run-netns\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403135    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-wtmp\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403171    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403177    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-textfile\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403202    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6679db18-7f2c-4478-9eb1-0a20b16040e4-secret-telemeter-client\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403202    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-tls\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403238    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-sys\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403258    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403275    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403300    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403309    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-openvswitch\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403342    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-root\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403354    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403367    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-registration-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403392    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-kubelet\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403450    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403760    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-registration-dir\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403759    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-tls\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403810    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-node-log\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403842    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-host-kubelet\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403876    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62d2b800-0c2b-49ca-906e-ec29326e1726-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403935    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-metrics-client-ca\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403970    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.403979    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-node-log\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404021    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-socket-dir-parent\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404176    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-dbus\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404176    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-multus-socket-dir-parent\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.404208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404230    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/274de09e-a693-46cf-b317-a0cbdfcf79df-metrics-client-ca\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404249    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404305    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-k8s-cni-cncf-io\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.404323    5244 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404403    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8541995b-77c8-4973-9736-f750edf753cb-dbus\") pod \"global-pull-secret-syncer-s77gw\" (UID: \"8541995b-77c8-4973-9736-f750edf753cb\") " pod="kube-system/global-pull-secret-syncer-s77gw"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404462    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-run-k8s-cni-cncf-io\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404513    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-systemd\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404550    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-systemd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404585    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-multus\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404601    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-systemd\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404618    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-etc-kubernetes\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404644    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404648    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae80c711-2bb7-43e4-9bf2-521dbc9e4291-run-systemd\") pod \"ovnkube-node-gmxs8\" (UID: \"ae80c711-2bb7-43e4-9bf2-521dbc9e4291\") " pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404677    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-sys-fs\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.404700    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.904679477 +0000 UTC m=+1.888676028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404735    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-sys-fs\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.404792    5244 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404797    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-multus\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.404855    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.904840543 +0000 UTC m=+1.888837095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404858    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-etc-kubernetes\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404736    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404929    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdtj\" (UniqueName: \"kubernetes.io/projected/62d2b800-0c2b-49ca-906e-ec29326e1726-kube-api-access-5wdtj\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.404974    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-system-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405000    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa443bcf-d427-46f2-b924-beb20bd4df7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jgqtm\" (UID: \"aa443bcf-d427-46f2-b924-beb20bd4df7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jgqtm"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405000    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-sys\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405052    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-serving-certs-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405081    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405102    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-run\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405129    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-bin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405157    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405238    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405388    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-conf\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405440    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-d\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405500    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-system-cni-dir\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405564    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-etc-sysctl-conf\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405566    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-sys\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405772    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6679db18-7f2c-4478-9eb1-0a20b16040e4-serving-certs-ca-bundle\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405856    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3648276-55d9-46b7-ac52-ea7223dd5868-run\") pod \"tuned-zv8ps\" (UID: \"d3648276-55d9-46b7-ac52-ea7223dd5868\") " pod="openshift-cluster-node-tuning-operator/tuned-zv8ps"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405907    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb673cf5-6137-4a72-8910-e957e3685a71-host-var-lib-cni-bin\") pod \"multus-gg868\" (UID: \"eb673cf5-6137-4a72-8910-e957e3685a71\") " pod="openshift-multus/multus-gg868"
Nov 03 21:15:30.406502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.405895    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5718e9a0-0924-41fe-a482-557054019ca9-system-cni-dir\") pod \"multus-additional-cni-plugins-fp2r2\" (UID: \"5718e9a0-0924-41fe-a482-557054019ca9\") " pod="openshift-multus/multus-additional-cni-plugins-fp2r2"
Nov 03 21:15:30.458927 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.458893    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlp9\" (UniqueName: \"kubernetes.io/projected/274de09e-a693-46cf-b317-a0cbdfcf79df-kube-api-access-fzlp9\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.459057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.459019    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc76r\" (UniqueName: \"kubernetes.io/projected/6679db18-7f2c-4478-9eb1-0a20b16040e4-kube-api-access-qc76r\") pod \"telemeter-client-964cff64-qf6zt\" (UID: \"6679db18-7f2c-4478-9eb1-0a20b16040e4\") " pod="openshift-monitoring/telemeter-client-964cff64-qf6zt"
Nov 03 21:15:30.459872 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.459849    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdtj\" (UniqueName: \"kubernetes.io/projected/62d2b800-0c2b-49ca-906e-ec29326e1726-kube-api-access-5wdtj\") pod \"monitoring-plugin-77f4f76594-gv74f\" (UID: \"62d2b800-0c2b-49ca-906e-ec29326e1726\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:30.460461 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.460445    5244 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 21:15:30.460525 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.460467    5244 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 21:15:30.460525 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.460479    5244 projected.go:196] Error preparing data for projected volume kube-api-access-fgtsb for pod openshift-console/console-78789694d8-xqjq4: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:30.460588 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.460556    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:30.960541312 +0000 UTC m=+1.944537865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fgtsb" (UniqueName: "kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:30.505750 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.505719    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-wtmp\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.505750 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.505757    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-sys\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.505933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.505774    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-root\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.505933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.505810    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-node-exporter-wtmp\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.505933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.505874    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-sys\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.505933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.505905    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/274de09e-a693-46cf-b317-a0cbdfcf79df-root\") pod \"node-exporter-xr2gw\" (UID: \"274de09e-a693-46cf-b317-a0cbdfcf79df\") " pod="openshift-monitoring/node-exporter-xr2gw"
Nov 03 21:15:30.739090 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.738980    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6jq4g"
Nov 03 21:15:30.910565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.910529    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.910565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.910566    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.910565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.910582    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.910603    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.910657    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:30.910681    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910655    5244 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910713    5244 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910736    5244 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910681    5244 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910754    5244 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910765    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:31.910751208 +0000 UTC m=+2.894747745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910681    5244 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910795    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:31.910773056 +0000 UTC m=+2.894769593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910818    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:31.910800324 +0000 UTC m=+2.894796875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910834    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:31.910827313 +0000 UTC m=+2.894823901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910844    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:31.910838521 +0000 UTC m=+2.894835058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"service-ca" not registered
Nov 03 21:15:30.910879 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:30.910853    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:31.910848931 +0000 UTC m=+2.894845469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-config" not registered
Nov 03 21:15:31.011786 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.011702    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.011928 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.011831    5244 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 21:15:31.011928 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.011845    5244 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 21:15:31.011928 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.011854    5244 projected.go:196] Error preparing data for projected volume kube-api-access-fgtsb for pod openshift-console/console-78789694d8-xqjq4: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:31.011928 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.011911    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:32.011897344 +0000 UTC m=+2.995893881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgtsb" (UniqueName: "kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:31.137372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.137340    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:31.657061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.657024    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:15:31.920756 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.920675    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.920756 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.920716    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.920756 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.920753    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.921020 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.920773    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.921020 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.920789    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.921114 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921044    5244 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:31.921239 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:31.921201    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:31.921296 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921208    5244 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 21:15:31.921375 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921321    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:33.92127587 +0000 UTC m=+4.905272412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:31.921375 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921239    5244 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:31.921375 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921363    5244 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:31.921510 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921390    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:33.921363555 +0000 UTC m=+4.905360091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"service-ca" not registered
Nov 03 21:15:31.921510 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921259    5244 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 21:15:31.921510 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921415    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:33.921401569 +0000 UTC m=+4.905398213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:31.921510 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921456    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:33.921436268 +0000 UTC m=+4.905432809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-config" not registered
Nov 03 21:15:31.921510 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921478    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:33.921466411 +0000 UTC m=+4.905462982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:31.921794 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921658    5244 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:31.921794 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:31.921768    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:33.921740903 +0000 UTC m=+4.905737472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:32.022655 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:32.022610    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:32.022812 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:32.022748    5244 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 21:15:32.022812 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:32.022762    5244 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 21:15:32.022812 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:32.022772    5244 projected.go:196] Error preparing data for projected volume kube-api-access-fgtsb for pod openshift-console/console-78789694d8-xqjq4: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:32.022915 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:32.022863    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:34.022850386 +0000 UTC m=+5.006846923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgtsb" (UniqueName: "kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:33.941503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:33.941457    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:33.941546    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:33.941582    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941616    5244 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:33.941640    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:33.941667    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:33.941688    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941707    5244 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941729    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:37.94171299 +0000 UTC m=+8.925709527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941732    5244 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941774    5244 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941778    5244 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941784    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:37.941765705 +0000 UTC m=+8.925762262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-config" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941802    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:37.941796107 +0000 UTC m=+8.925792643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941799    5244 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941812    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:37.941807407 +0000 UTC m=+8.925803944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941826    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:37.941821731 +0000 UTC m=+8.925818269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"service-ca" not registered
Nov 03 21:15:33.941944 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:33.941878    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:37.941862732 +0000 UTC m=+8.925859286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:34.043010 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:34.042975    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:34.043179 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:34.043104    5244 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 21:15:34.043179 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:34.043118    5244 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 21:15:34.043179 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:34.043127    5244 projected.go:196] Error preparing data for projected volume kube-api-access-fgtsb for pod openshift-console/console-78789694d8-xqjq4: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:34.043179 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:34.043175    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:38.043163578 +0000 UTC m=+9.027160127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgtsb" (UniqueName: "kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:37.307862 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.307789    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h4zdr"
Nov 03 21:15:37.979813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.979773    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:37.979813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.979819    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.979869    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.979888    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.979906    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:37.979934    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.979939    5244 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980004    5244 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980014    5244 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980024    5244 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980041    5244 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980050    5244 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980021    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:45.980000023 +0000 UTC m=+16.963996575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980084    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:45.980069826 +0000 UTC m=+16.964066368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 21:15:37.980098 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980103    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:45.980090014 +0000 UTC m=+16.964086553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"trusted-ca-bundle" not registered
Nov 03 21:15:37.980526 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980125    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:45.980114127 +0000 UTC m=+16.964110665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"console-config" not registered
Nov 03 21:15:37.980526 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980140    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:45.980131716 +0000 UTC m=+16.964128258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 21:15:37.980526 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:37.980156    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:45.980147761 +0000 UTC m=+16.964144336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : object "openshift-console"/"service-ca" not registered
Nov 03 21:15:38.081423 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:38.081373    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:38.081580 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:38.081564    5244 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 21:15:38.081615 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:38.081582    5244 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 21:15:38.081615 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:38.081594    5244 projected.go:196] Error preparing data for projected volume kube-api-access-fgtsb for pod openshift-console/console-78789694d8-xqjq4: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:38.081700 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:38.081652    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb podName:e327fb0c-d657-4d21-862d-de5f5fd7dbd3 nodeName:}" failed. No retries permitted until 2026-11-03 21:15:46.08163586 +0000 UTC m=+17.065632400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fgtsb" (UniqueName: "kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb") pod "console-78789694d8-xqjq4" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 21:15:38.188158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:38.188119    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:38.192730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:38.192702    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gv74f"
Nov 03 21:15:38.396089 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:38.395999    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:38.396553 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:38.396200    5244 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 21:15:38.876148 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:38.876114    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gmxs8"
Nov 03 21:15:39.652967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:39.652936    5244 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-65.us-west-1.compute.internal" event="NodeReady"
Nov 03 21:15:45.624199 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.624163    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f4674686d-tggpk"]
Nov 03 21:15:45.627128 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.627111    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.679842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.679810    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 21:15:45.728904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.728878    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 21:15:45.729080 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.729063    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 21:15:45.729427 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.729409    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 21:15:45.729518 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.729414    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 21:15:45.748765 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.748736    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-console-config\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.748905 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.748884    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76028b57-87d6-4730-82f4-904da75b73ba-console-oauth-config\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.748975 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.748960    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-oauth-serving-cert\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.749027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.749014    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76028b57-87d6-4730-82f4-904da75b73ba-tmp\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.749059 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.749035    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-trusted-ca-bundle\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.749094 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.749069    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctwr\" (UniqueName: \"kubernetes.io/projected/76028b57-87d6-4730-82f4-904da75b73ba-kube-api-access-tctwr\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.749148 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.749128    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76028b57-87d6-4730-82f4-904da75b73ba-console-serving-cert\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.749205 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.749194    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-service-ca\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.778588 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.778552    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 21:15:45.778948 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.778930    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 21:15:45.828870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.828836    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kz2rr\""
Nov 03 21:15:45.829495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.829472    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 21:15:45.830032 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.830011    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4674686d-tggpk"]
Nov 03 21:15:45.849778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849757    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-service-ca\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.849849 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849786    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-console-config\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.849849 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849842    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76028b57-87d6-4730-82f4-904da75b73ba-console-oauth-config\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.849926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849864    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-oauth-serving-cert\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.849926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849897    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76028b57-87d6-4730-82f4-904da75b73ba-tmp\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.849926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849922    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-trusted-ca-bundle\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849961    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctwr\" (UniqueName: \"kubernetes.io/projected/76028b57-87d6-4730-82f4-904da75b73ba-kube-api-access-tctwr\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.849988    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76028b57-87d6-4730-82f4-904da75b73ba-console-serving-cert\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.850270    5244 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 21:15:45.850402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.850358    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76028b57-87d6-4730-82f4-904da75b73ba-tmp\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850699 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.850657    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-service-ca\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850699 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.850688    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-oauth-serving-cert\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.850781    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-console-config\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.850863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.850825    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76028b57-87d6-4730-82f4-904da75b73ba-trusted-ca-bundle\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.852672 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.852655    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/76028b57-87d6-4730-82f4-904da75b73ba-console-serving-cert\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.852728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.852685    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/76028b57-87d6-4730-82f4-904da75b73ba-console-oauth-config\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.906451 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.906380    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctwr\" (UniqueName: \"kubernetes.io/projected/76028b57-87d6-4730-82f4-904da75b73ba-kube-api-access-tctwr\") pod \"console-f4674686d-tggpk\" (UID: \"76028b57-87d6-4730-82f4-904da75b73ba\") " pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:45.937146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:45.937107    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:46.052080 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052037    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052103    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052126    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052146    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052167    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052225    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052366    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052387    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052443    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052561 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052512    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052554    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.052774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.052758    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.152750 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.152697    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.153434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.153415    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"console-78789694d8-xqjq4\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") " pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:46.304698 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:46.304664    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4674686d-tggpk"]
Nov 03 21:15:46.308750 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:46.308707    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76028b57_87d6_4730_82f4_904da75b73ba.slice/crio-0da9be38075cd75e3ec321ba096dd2bdfb8be7455ffd7ac08a068b3019dd013d WatchSource:0}: Error finding container 0da9be38075cd75e3ec321ba096dd2bdfb8be7455ffd7ac08a068b3019dd013d: Status 404 returned error can't find the container with id 0da9be38075cd75e3ec321ba096dd2bdfb8be7455ffd7ac08a068b3019dd013d
Nov 03 21:15:47.251857 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:47.251820    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4674686d-tggpk" event={"ID":"76028b57-87d6-4730-82f4-904da75b73ba","Type":"ContainerStarted","Data":"aae0edc5ee109a75ef0e80fc1bc3d9a688e57de2501a30b2a8205bb11ab7d27c"}
Nov 03 21:15:47.252472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:47.251873    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4674686d-tggpk" event={"ID":"76028b57-87d6-4730-82f4-904da75b73ba","Type":"ContainerStarted","Data":"0da9be38075cd75e3ec321ba096dd2bdfb8be7455ffd7ac08a068b3019dd013d"}
Nov 03 21:15:47.415379 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:47.415283    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f4674686d-tggpk" podStartSLOduration=2.41526766 podStartE2EDuration="2.41526766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:47.36590424 +0000 UTC m=+18.349900798" watchObservedRunningTime="2026-11-03 21:15:47.41526766 +0000 UTC m=+18.399264216"
Nov 03 21:15:52.435763 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:52.435725    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:15:53.897448 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.897411    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 21:15:53.900546 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.900524    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:53.946973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.946945    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 21:15:53.988822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.988783    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 21:15:53.988822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.988789    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 21:15:53.988822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.988825    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 21:15:53.989077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.988879    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ccvj\""
Nov 03 21:15:53.989077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.988895    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 21:15:53.989077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:53.988935    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 21:15:54.002220 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.002199    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 21:15:54.002370 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.002353    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 21:15:54.002428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.002416    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 21:15:54.009714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009691    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009784 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009731    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-config-volume\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009784 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009761    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00d3f9c0-2f17-4adb-b15d-246be24d4117-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009858    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009904    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00d3f9c0-2f17-4adb-b15d-246be24d4117-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009922    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56mfc\" (UniqueName: \"kubernetes.io/projected/00d3f9c0-2f17-4adb-b15d-246be24d4117-kube-api-access-56mfc\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009943    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.009976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009963    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-web-config\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.010038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.009981    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.010080 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.010063    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00d3f9c0-2f17-4adb-b15d-246be24d4117-config-out\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.010115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.010083    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00d3f9c0-2f17-4adb-b15d-246be24d4117-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.010115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.010103    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00d3f9c0-2f17-4adb-b15d-246be24d4117-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.010206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.010122    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.052203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.052170    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 21:15:54.111472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111436    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-web-config\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.111472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111473    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.111688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111504    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00d3f9c0-2f17-4adb-b15d-246be24d4117-config-out\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.111737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111680    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00d3f9c0-2f17-4adb-b15d-246be24d4117-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.111737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111719    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00d3f9c0-2f17-4adb-b15d-246be24d4117-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.111844 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111752    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.111957    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112011    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-config-volume\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112048    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00d3f9c0-2f17-4adb-b15d-246be24d4117-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112090    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00d3f9c0-2f17-4adb-b15d-246be24d4117-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112087    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112174    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00d3f9c0-2f17-4adb-b15d-246be24d4117-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112200    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56mfc\" (UniqueName: \"kubernetes.io/projected/00d3f9c0-2f17-4adb-b15d-246be24d4117-kube-api-access-56mfc\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.112295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.112238    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.113193 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.113169    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00d3f9c0-2f17-4adb-b15d-246be24d4117-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.113621 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.113531    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00d3f9c0-2f17-4adb-b15d-246be24d4117-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.114959 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.114931    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.115570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.115524    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-web-config\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.115671 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.115641    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.116064 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.116018    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.116385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.116365    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.116470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.116411    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00d3f9c0-2f17-4adb-b15d-246be24d4117-config-out\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.116470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.116462    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-config-volume\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.116593 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.116493    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00d3f9c0-2f17-4adb-b15d-246be24d4117-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.117161 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.117141    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00d3f9c0-2f17-4adb-b15d-246be24d4117-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.169843 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.169770    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56mfc\" (UniqueName: \"kubernetes.io/projected/00d3f9c0-2f17-4adb-b15d-246be24d4117-kube-api-access-56mfc\") pod \"alertmanager-main-0\" (UID: \"00d3f9c0-2f17-4adb-b15d-246be24d4117\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.216693 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.216655    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 21:15:54.643144 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:54.643105    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 21:15:54.648702 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:15:54.648665    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d3f9c0_2f17_4adb_b15d_246be24d4117.slice/crio-f3a64eea5fd2e97d0e5678f14e4c81d1d4a8f495a58339e1bee576fd3bebcd4b WatchSource:0}: Error finding container f3a64eea5fd2e97d0e5678f14e4c81d1d4a8f495a58339e1bee576fd3bebcd4b: Status 404 returned error can't find the container with id f3a64eea5fd2e97d0e5678f14e4c81d1d4a8f495a58339e1bee576fd3bebcd4b
Nov 03 21:15:55.222124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.222070    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-78789694d8-xqjq4" podUID="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" containerName="console" containerID="cri-o://191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5" gracePeriod=15
Nov 03 21:15:55.283762 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.283723    5244 generic.go:356] "Generic (PLEG): container finished" podID="00d3f9c0-2f17-4adb-b15d-246be24d4117" containerID="68c7b708c77e9459267a184b93b81a3f4e4d08d115a2af5abc7e5fab7dce0254" exitCode=0
Nov 03 21:15:55.283934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.283781    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerDied","Data":"68c7b708c77e9459267a184b93b81a3f4e4d08d115a2af5abc7e5fab7dce0254"}
Nov 03 21:15:55.283934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.283834    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"f3a64eea5fd2e97d0e5678f14e4c81d1d4a8f495a58339e1bee576fd3bebcd4b"}
Nov 03 21:15:55.285015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.284998    5244 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:15:55.493503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.493466    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:55.626286 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626253    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626299    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626383    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626417    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626589 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626473    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626589 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626499    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626589 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626548    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626595    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") pod \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\" (UID: \"e327fb0c-d657-4d21-862d-de5f5fd7dbd3\") "
Nov 03 21:15:55.626719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626635    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:55.626820 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626797    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:55.626871 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626821    5244 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.626871 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626843    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:55.626990 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.626967    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:15:55.627080 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.627061    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:15:55.629155 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.629124    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:55.629283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.629176    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "kube-api-access-fgtsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:15:55.629283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.629185    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert" pod "e327fb0c-d657-4d21-862d-de5f5fd7dbd3" (UID: "e327fb0c-d657-4d21-862d-de5f5fd7dbd3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:15:55.727644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727551    5244 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-serving-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.727644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727588    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtsb\" (UniqueName: \"kubernetes.io/projected/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-kube-api-access-fgtsb\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.727644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727604    5244 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-trusted-ca-bundle\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.727644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727616    5244 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-oauth-serving-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.727644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727632    5244 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-tmp\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.727644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727644    5244 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-config\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.727945 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.727658    5244 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e327fb0c-d657-4d21-862d-de5f5fd7dbd3-console-oauth-config\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:15:55.937805 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.937769    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:55.938015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.937838    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:55.942659 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:55.942635    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:56.288609 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.288576    5244 generic.go:356] "Generic (PLEG): container finished" podID="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" containerID="191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5" exitCode=0
Nov 03 21:15:56.289065 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.288627    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78789694d8-xqjq4"
Nov 03 21:15:56.289065 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.288668    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78789694d8-xqjq4" event={"ID":"e327fb0c-d657-4d21-862d-de5f5fd7dbd3","Type":"ContainerDied","Data":"191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5"}
Nov 03 21:15:56.289065 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.288734    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78789694d8-xqjq4" event={"ID":"e327fb0c-d657-4d21-862d-de5f5fd7dbd3","Type":"ContainerDied","Data":"4cd6bdbccbf82083c6028045ea643dca448cf2e703198a9546dcd08f8c3bda53"}
Nov 03 21:15:56.289065 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.288764    5244 scope.go:122] "RemoveContainer" containerID="191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5"
Nov 03 21:15:56.292705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.292684    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f4674686d-tggpk"
Nov 03 21:15:56.297470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.297453    5244 scope.go:122] "RemoveContainer" containerID="191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5"
Nov 03 21:15:56.297742 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:15:56.297723    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5\": container with ID starting with 191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5 not found: ID does not exist" containerID="191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5"
Nov 03 21:15:56.297782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.297752    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5"} err="failed to get container status \"191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5\": rpc error: code = NotFound desc = could not find container \"191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5\": container with ID starting with 191d03c167315b93ee776a391bac91766153c34cfa149a0bfb3f58cd2bc6c5b5 not found: ID does not exist"
Nov 03 21:15:56.451295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.451260    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78789694d8-xqjq4"]
Nov 03 21:15:56.503510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:56.503472    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78789694d8-xqjq4"]
Nov 03 21:15:57.134047 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:57.134014    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" path="/var/lib/kubelet/pods/e327fb0c-d657-4d21-862d-de5f5fd7dbd3/volumes"
Nov 03 21:15:58.298809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:58.298783    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"0a94e152d33fef652ec28bce7fa953b0ff4d882c057d88afe0dd76d7f797cb1f"}
Nov 03 21:15:59.305354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:59.305299    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"bcb8b2f0a7582b032ffd231cb75888ff62481b5a2734e7a1cc51a18373267de5"}
Nov 03 21:15:59.305773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:59.305401    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"eb7223d625e752d29a04408d0c25708f3e96fb6eb0b60fc762a678494c9bde8c"}
Nov 03 21:15:59.305773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:59.305440    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"e48fefba3e814b4a315cd6d2d04efe82796b49dfed5be5e877b0a3c91d11b467"}
Nov 03 21:15:59.305773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:15:59.305478    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"b4502a423cc6ad99e37d4e9c37fc8e71c764d9c850db3d899264aa050627d9f1"}
Nov 03 21:16:01.314763 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:16:01.314727    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00d3f9c0-2f17-4adb-b15d-246be24d4117","Type":"ContainerStarted","Data":"32ed94d5fe200224e8c5626fb39c8b886c4193b9f2e0f9f63dfdb62ec4840af9"}
Nov 03 21:16:01.581842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:16:01.581729    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.008516115 podStartE2EDuration="8.581714756s" totalImagesPullingTime="4.556903669s" totalInitContainerRuntime="16.294972ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:01.532064065 +0000 UTC m=+32.516060665" watchObservedRunningTime="2026-11-03 21:16:01.581714756 +0000 UTC m=+32.565711379"
Nov 03 21:16:01.659536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:16:01.659506    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kgjns"
Nov 03 21:16:22.435563 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:16:22.435523    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:16:52.435644 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:16:52.435606    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:17:22.436754 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:17:22.436669    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:17:52.436630 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:17:52.436592    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:18:22.435477 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:18:22.435442    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:18:33.550811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.550776    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"]
Nov 03 21:18:33.551307 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.551055    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" containerName="console"
Nov 03 21:18:33.551307 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.551067    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" containerName="console"
Nov 03 21:18:33.551307 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.551133    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e327fb0c-d657-4d21-862d-de5f5fd7dbd3" containerName="console"
Nov 03 21:18:33.554036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.554020    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"]
Nov 03 21:18:33.554166 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.554145    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.556970 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.556954    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.617179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.617151    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 21:18:33.617389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.617243    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 21:18:33.664624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.664588    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-xhsmb\""
Nov 03 21:18:33.664624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.664588    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2l2wp\""
Nov 03 21:18:33.665167 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.665153    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"]
Nov 03 21:18:33.665572 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.665554    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"]
Nov 03 21:18:33.706273 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.706239    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.706462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.706293    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.706462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.706374    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllpx\" (UniqueName: \"kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.706548 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.706513    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.706582 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.706551    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.706643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.706628    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsbt\" (UniqueName: \"kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.807055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.806957    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.807055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807002    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.807055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807044    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsbt\" (UniqueName: \"kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.807433 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807082    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.807433 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807125    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.807433 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807146    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllpx\" (UniqueName: \"kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.807433 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807409    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.807567 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807436    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.807567 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807516    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.807567 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.807529    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.864077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.864038    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsbt\" (UniqueName: \"kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.864252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.864229    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllpx\" (UniqueName: \"kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.870917 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.870895    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:33.876911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.876895    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:33.998296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:33.998266    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"]
Nov 03 21:18:34.001815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.001796    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.054245 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054190    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 21:18:34.054245 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054214    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 21:18:34.054245 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054214    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 21:18:34.054245 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054216    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s9fpr\""
Nov 03 21:18:34.054576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054264    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 21:18:34.054576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054251    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 21:18:34.054576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.054246    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 21:18:34.064952 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.064879    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"]
Nov 03 21:18:34.109215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109168    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.109215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109217    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.109496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109247    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.109496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109374    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.109496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109409    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhzl\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.109496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109435    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.109496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.109486    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210039 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210002    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210039 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210044    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhzl\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210069    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210093    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210163    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210186    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210218    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.210865 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.210842    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.212429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.212411    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.212669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.212630    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.212873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.212855    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.212978 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.212958    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.247667 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.247632    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"]
Nov 03 21:18:34.251094 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:18:34.251066    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8111d594_f203_43e5_b242_781f5e78c538.slice/crio-dca2d9fec3e33f8fd615bceef1cb1a0582751863adf1b25a74fa91d50a933b8d WatchSource:0}: Error finding container dca2d9fec3e33f8fd615bceef1cb1a0582751863adf1b25a74fa91d50a933b8d: Status 404 returned error can't find the container with id dca2d9fec3e33f8fd615bceef1cb1a0582751863adf1b25a74fa91d50a933b8d
Nov 03 21:18:34.266976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.266947    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhzl\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.267354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.267319    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2s5t5\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.299551 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.299523    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"]
Nov 03 21:18:34.302887 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:18:34.302861    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d631a36_5ba6_4521_bf88_a27fdf97d4e9.slice/crio-d1aab5deb07e841aa5ee2c4f5cad37cbda4ca75df024cbe0ff5f045e24249fa5 WatchSource:0}: Error finding container d1aab5deb07e841aa5ee2c4f5cad37cbda4ca75df024cbe0ff5f045e24249fa5: Status 404 returned error can't find the container with id d1aab5deb07e841aa5ee2c4f5cad37cbda4ca75df024cbe0ff5f045e24249fa5
Nov 03 21:18:34.314043 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.314017    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:34.685255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.685228    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"]
Nov 03 21:18:34.688917 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:18:34.688890    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e7a1f8_ed50_4de4_b02d_7847c2f16352.slice/crio-7bff5c51058afca3fb6deaf465a58a57a6818dc7fa83fb10f08bce5237fff3c9 WatchSource:0}: Error finding container 7bff5c51058afca3fb6deaf465a58a57a6818dc7fa83fb10f08bce5237fff3c9: Status 404 returned error can't find the container with id 7bff5c51058afca3fb6deaf465a58a57a6818dc7fa83fb10f08bce5237fff3c9
Nov 03 21:18:34.819683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.819642    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" event={"ID":"12e7a1f8-ed50-4de4-b02d-7847c2f16352","Type":"ContainerStarted","Data":"7bff5c51058afca3fb6deaf465a58a57a6818dc7fa83fb10f08bce5237fff3c9"}
Nov 03 21:18:34.820752 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.820724    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" event={"ID":"2d631a36-5ba6-4521-bf88-a27fdf97d4e9","Type":"ContainerStarted","Data":"d1aab5deb07e841aa5ee2c4f5cad37cbda4ca75df024cbe0ff5f045e24249fa5"}
Nov 03 21:18:34.823359 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:34.822207    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" event={"ID":"8111d594-f203-43e5-b242-781f5e78c538","Type":"ContainerStarted","Data":"dca2d9fec3e33f8fd615bceef1cb1a0582751863adf1b25a74fa91d50a933b8d"}
Nov 03 21:18:38.783907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:38.783217    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:18:38.783907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:38.783292    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:18:39.847966 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:39.847927    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" event={"ID":"12e7a1f8-ed50-4de4-b02d-7847c2f16352","Type":"ContainerStarted","Data":"d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3"}
Nov 03 21:18:39.848525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:39.848297    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:39.850276 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:39.850248    5244 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2s5t5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:18:39.850428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:39.850317    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:18:40.053038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:40.052968    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" podStartSLOduration=2.961815838 podStartE2EDuration="7.052945129s" totalImagesPullingTime="4.091129291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:40.012529819 +0000 UTC m=+190.996526397" watchObservedRunningTime="2026-11-03 21:18:40.052945129 +0000 UTC m=+191.036941712"
Nov 03 21:18:40.853745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:40.853711    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:18:42.860692 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:42.860649    5244 generic.go:356] "Generic (PLEG): container finished" podID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerID="4f40a8e77dda19dc56b88aa88dd234a283e207fa6e0daca51499d00ad6473032" exitCode=0
Nov 03 21:18:42.861161 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:42.860739    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" event={"ID":"2d631a36-5ba6-4521-bf88-a27fdf97d4e9","Type":"ContainerDied","Data":"4f40a8e77dda19dc56b88aa88dd234a283e207fa6e0daca51499d00ad6473032"}
Nov 03 21:18:42.862254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:42.862230    5244 generic.go:356] "Generic (PLEG): container finished" podID="8111d594-f203-43e5-b242-781f5e78c538" containerID="a74197b1c6282385e68006347169ae70f10596e547915402154eda11f5f850e4" exitCode=0
Nov 03 21:18:42.862321 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:42.862272    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" event={"ID":"8111d594-f203-43e5-b242-781f5e78c538","Type":"ContainerDied","Data":"a74197b1c6282385e68006347169ae70f10596e547915402154eda11f5f850e4"}
Nov 03 21:18:46.878746 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:46.878708    5244 generic.go:356] "Generic (PLEG): container finished" podID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerID="b2b84365780e7c73a83faab0015f70843d04b1c5a292214c8621953f75af181d" exitCode=0
Nov 03 21:18:46.879125 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:46.878788    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" event={"ID":"2d631a36-5ba6-4521-bf88-a27fdf97d4e9","Type":"ContainerDied","Data":"b2b84365780e7c73a83faab0015f70843d04b1c5a292214c8621953f75af181d"}
Nov 03 21:18:51.901429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:51.901347    5244 generic.go:356] "Generic (PLEG): container finished" podID="8111d594-f203-43e5-b242-781f5e78c538" containerID="6605b6ac9f7d69443f725a70fcff62ddf32c946f7f0516d70351a0f15a2477b1" exitCode=0
Nov 03 21:18:51.901881 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:51.901432    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" event={"ID":"8111d594-f203-43e5-b242-781f5e78c538","Type":"ContainerDied","Data":"6605b6ac9f7d69443f725a70fcff62ddf32c946f7f0516d70351a0f15a2477b1"}
Nov 03 21:18:52.438780 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:18:52.438712    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:18:54.916459 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:54.916377    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" event={"ID":"2d631a36-5ba6-4521-bf88-a27fdf97d4e9","Type":"ContainerStarted","Data":"7bdf842d764f8c244bd92e05ccd282f7fcca4e8ae110b20670d8c51b4b1b3a2e"}
Nov 03 21:18:55.030236 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:55.030171    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" podStartSLOduration=3.231262038 podStartE2EDuration="22.030156403s" totalImagesPullingTime="18.798894365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:55.02984287 +0000 UTC m=+206.013839449" watchObservedRunningTime="2026-11-03 21:18:55.030156403 +0000 UTC m=+206.014152961"
Nov 03 21:18:55.922775 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:55.922743    5244 generic.go:356] "Generic (PLEG): container finished" podID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerID="7bdf842d764f8c244bd92e05ccd282f7fcca4e8ae110b20670d8c51b4b1b3a2e" exitCode=0
Nov 03 21:18:55.923381 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:55.922836    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" event={"ID":"2d631a36-5ba6-4521-bf88-a27fdf97d4e9","Type":"ContainerDied","Data":"7bdf842d764f8c244bd92e05ccd282f7fcca4e8ae110b20670d8c51b4b1b3a2e"}
Nov 03 21:18:56.928254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:56.928218    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" event={"ID":"8111d594-f203-43e5-b242-781f5e78c538","Type":"ContainerStarted","Data":"acac2ac1798b7798b9d877155824a7a1f52ac95f52b343b9e58200f1a3da4fd1"}
Nov 03 21:18:57.041638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.041584    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" podStartSLOduration=2.977644658 podStartE2EDuration="24.041564286s" totalImagesPullingTime="21.063919628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:57.041050392 +0000 UTC m=+208.025046970" watchObservedRunningTime="2026-11-03 21:18:57.041564286 +0000 UTC m=+208.025560846"
Nov 03 21:18:57.073604 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.073574    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:57.124291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.124254    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx\" (UniqueName: \"kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx\") pod \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") "
Nov 03 21:18:57.124495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.124307    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util\") pod \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") "
Nov 03 21:18:57.124495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.124423    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle\") pod \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\" (UID: \"2d631a36-5ba6-4521-bf88-a27fdf97d4e9\") "
Nov 03 21:18:57.125248 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.125221    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle" pod "2d631a36-5ba6-4521-bf88-a27fdf97d4e9" (UID: "2d631a36-5ba6-4521-bf88-a27fdf97d4e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:57.127194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.127169    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx" pod "2d631a36-5ba6-4521-bf88-a27fdf97d4e9" (UID: "2d631a36-5ba6-4521-bf88-a27fdf97d4e9"). InnerVolumeSpecName "kube-api-access-tllpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:57.131039 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.131010    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util" pod "2d631a36-5ba6-4521-bf88-a27fdf97d4e9" (UID: "2d631a36-5ba6-4521-bf88-a27fdf97d4e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:57.225477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.225441    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tllpx\" (UniqueName: \"kubernetes.io/projected/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-kube-api-access-tllpx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:18:57.225477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.225473    5244 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-util\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:18:57.225477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.225483    5244 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d631a36-5ba6-4521-bf88-a27fdf97d4e9-bundle\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:18:57.933724 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.933685    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz" event={"ID":"2d631a36-5ba6-4521-bf88-a27fdf97d4e9","Type":"ContainerDied","Data":"d1aab5deb07e841aa5ee2c4f5cad37cbda4ca75df024cbe0ff5f045e24249fa5"}
Nov 03 21:18:57.934158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.933742    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1aab5deb07e841aa5ee2c4f5cad37cbda4ca75df024cbe0ff5f045e24249fa5"
Nov 03 21:18:57.934158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.933701    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjpfcz"
Nov 03 21:18:57.936126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.936092    5244 generic.go:356] "Generic (PLEG): container finished" podID="8111d594-f203-43e5-b242-781f5e78c538" containerID="acac2ac1798b7798b9d877155824a7a1f52ac95f52b343b9e58200f1a3da4fd1" exitCode=0
Nov 03 21:18:57.936259 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:57.936177    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" event={"ID":"8111d594-f203-43e5-b242-781f5e78c538","Type":"ContainerDied","Data":"acac2ac1798b7798b9d877155824a7a1f52ac95f52b343b9e58200f1a3da4fd1"}
Nov 03 21:18:59.068290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.068263    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:59.141459 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.141422    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle\") pod \"8111d594-f203-43e5-b242-781f5e78c538\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") "
Nov 03 21:18:59.141633 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.141478    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt\" (UniqueName: \"kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt\") pod \"8111d594-f203-43e5-b242-781f5e78c538\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") "
Nov 03 21:18:59.141633 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.141568    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util\") pod \"8111d594-f203-43e5-b242-781f5e78c538\" (UID: \"8111d594-f203-43e5-b242-781f5e78c538\") "
Nov 03 21:18:59.141903 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.141876    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle" pod "8111d594-f203-43e5-b242-781f5e78c538" (UID: "8111d594-f203-43e5-b242-781f5e78c538"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:59.143696 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.143668    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt" pod "8111d594-f203-43e5-b242-781f5e78c538" (UID: "8111d594-f203-43e5-b242-781f5e78c538"). InnerVolumeSpecName "kube-api-access-spsbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:18:59.146237 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.146216    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util" pod "8111d594-f203-43e5-b242-781f5e78c538" (UID: "8111d594-f203-43e5-b242-781f5e78c538"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:18:59.242766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.242681    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsbt\" (UniqueName: \"kubernetes.io/projected/8111d594-f203-43e5-b242-781f5e78c538-kube-api-access-spsbt\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:18:59.242766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.242712    5244 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-util\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:18:59.242766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.242726    5244 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8111d594-f203-43e5-b242-781f5e78c538-bundle\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:18:59.945004 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.944974    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk" event={"ID":"8111d594-f203-43e5-b242-781f5e78c538","Type":"ContainerDied","Data":"dca2d9fec3e33f8fd615bceef1cb1a0582751863adf1b25a74fa91d50a933b8d"}
Nov 03 21:18:59.945004 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.944993    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zpfxk"
Nov 03 21:18:59.945210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:18:59.945019    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dca2d9fec3e33f8fd615bceef1cb1a0582751863adf1b25a74fa91d50a933b8d"
Nov 03 21:19:15.064964 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.064927    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"]
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065195    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="pull"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065206    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="pull"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065213    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="extract"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065219    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="extract"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065228    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="util"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065233    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="util"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065239    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="extract"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065244    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="extract"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065255    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="pull"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065260    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="pull"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065266    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="util"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065270    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="util"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065345    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d631a36-5ba6-4521-bf88-a27fdf97d4e9" containerName="extract"
Nov 03 21:19:15.065503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.065353    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8111d594-f203-43e5-b242-781f5e78c538" containerName="extract"
Nov 03 21:19:15.067151 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.067134    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.168357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.168299    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.168535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.168366    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdlp\" (UniqueName: \"kubernetes.io/projected/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-kube-api-access-ctdlp\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.168535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.168453    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.168535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.168523    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.168662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.168613    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.168662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.168659    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.194366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.194310    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 21:19:15.194562 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.194371    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 21:19:15.194778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.194758    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 21:19:15.194831 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.194797    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 21:19:15.194869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.194858    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tmvnd\""
Nov 03 21:19:15.195108 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.195095    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 21:19:15.269813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.269778    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.269984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.269822    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.269984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.269850    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.269984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.269877    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdlp\" (UniqueName: \"kubernetes.io/projected/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-kube-api-access-ctdlp\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.269984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.269909    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.269984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.269973    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.270506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.270475    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-manifests\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.272432 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.272409    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-tmp\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.272636 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.272618    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-metrics-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.272699 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.272675    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.272699 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.272679    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-webhook-cert\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.329575 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.329493    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"]
Nov 03 21:19:15.331906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.331880    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdlp\" (UniqueName: \"kubernetes.io/projected/0c34824d-c64e-4dd6-936f-cc7cfb8b7538-kube-api-access-ctdlp\") pod \"opendatahub-operator-controller-manager-5745664b78-jccv9\" (UID: \"0c34824d-c64e-4dd6-936f-cc7cfb8b7538\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.379665 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.379624    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:15.868820 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:15.868783    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"]
Nov 03 21:19:15.872758 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:19:15.872733    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c34824d_c64e_4dd6_936f_cc7cfb8b7538.slice/crio-41d96796be5ac1b97a37b5d7b36e5e0b721b963ff3fe75b38026e1fcfbb4240a WatchSource:0}: Error finding container 41d96796be5ac1b97a37b5d7b36e5e0b721b963ff3fe75b38026e1fcfbb4240a: Status 404 returned error can't find the container with id 41d96796be5ac1b97a37b5d7b36e5e0b721b963ff3fe75b38026e1fcfbb4240a
Nov 03 21:19:16.007491 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:16.007456    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9" event={"ID":"0c34824d-c64e-4dd6-936f-cc7cfb8b7538","Type":"ContainerStarted","Data":"41d96796be5ac1b97a37b5d7b36e5e0b721b963ff3fe75b38026e1fcfbb4240a"}
Nov 03 21:19:21.028516 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:21.028481    5244 generic.go:356] "Generic (PLEG): container finished" podID="0c34824d-c64e-4dd6-936f-cc7cfb8b7538" containerID="2e435c6281148cf266e7699b32abd2c0cc898a20f9d1e66b78d6ea2235447de8" exitCode=0
Nov 03 21:19:21.028910 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:21.028554    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9" event={"ID":"0c34824d-c64e-4dd6-936f-cc7cfb8b7538","Type":"ContainerDied","Data":"2e435c6281148cf266e7699b32abd2c0cc898a20f9d1e66b78d6ea2235447de8"}
Nov 03 21:19:22.034042 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:22.034006    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9" event={"ID":"0c34824d-c64e-4dd6-936f-cc7cfb8b7538","Type":"ContainerStarted","Data":"92389e11c76ba9d29a5c01c5aaab81187dfdc6194c8ede1430859518e95ab9eb"}
Nov 03 21:19:22.034586 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:22.034114    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:22.308405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:22.308351    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9" podStartSLOduration=2.820523083 podStartE2EDuration="8.308311405s" totalImagesPullingTime="5.021637359s" totalInitContainerRuntime="466.150963ms" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:22.25860209 +0000 UTC m=+233.242598670" watchObservedRunningTime="2026-11-03 21:19:22.308311405 +0000 UTC m=+233.292307964"
Nov 03 21:19:22.435589 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:19:22.435547    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:19:33.039838 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:33.039801    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5745664b78-jccv9"
Nov 03 21:19:52.436200 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:19:52.436159    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:19:52.454660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.454625    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-545b54f677-5rl57"]
Nov 03 21:19:52.459644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.459623    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.552580 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.552537    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545b54f677-5rl57"]
Nov 03 21:19:52.563339 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.563300    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsllj\""
Nov 03 21:19:52.563339 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.563315    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 21:19:52.563339 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.563319    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 21:19:52.589073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.589035    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-tmp\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.589290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.589085    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbws\" (UniqueName: \"kubernetes.io/projected/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-kube-api-access-trbws\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.589290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.589235    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-tls-certs\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.690261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.690171    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-tls-certs\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.690261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.690208    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-tmp\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.690261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.690247    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbws\" (UniqueName: \"kubernetes.io/projected/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-kube-api-access-trbws\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.692717 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.692690    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-tmp\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.692917 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.692897    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-tls-certs\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.753133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.753097    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbws\" (UniqueName: \"kubernetes.io/projected/0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b-kube-api-access-trbws\") pod \"kube-auth-proxy-545b54f677-5rl57\" (UID: \"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b\") " pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:52.769879 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:52.769850    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57"
Nov 03 21:19:53.139031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:53.138950    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-545b54f677-5rl57"]
Nov 03 21:19:53.143354 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:19:53.143310    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5f8c7f_c09b_4b32_bf94_c6343bb4ad3b.slice/crio-22748e303aa7c1dd456291578c1cee1863465850532c6dc51be2e19f2b846767 WatchSource:0}: Error finding container 22748e303aa7c1dd456291578c1cee1863465850532c6dc51be2e19f2b846767: Status 404 returned error can't find the container with id 22748e303aa7c1dd456291578c1cee1863465850532c6dc51be2e19f2b846767
Nov 03 21:19:53.149881 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:53.149851    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57" event={"ID":"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b","Type":"ContainerStarted","Data":"22748e303aa7c1dd456291578c1cee1863465850532c6dc51be2e19f2b846767"}
Nov 03 21:19:56.163311 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:56.163275    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57" event={"ID":"0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b","Type":"ContainerStarted","Data":"06abf49830b0f4085a54f782496bc264f48e960de99f8fd088a333f8bc06f7e2"}
Nov 03 21:19:56.277033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:19:56.276979    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-545b54f677-5rl57" podStartSLOduration=1.377044546 podStartE2EDuration="4.276962252s" totalImagesPullingTime="2.899917706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:56.276475155 +0000 UTC m=+267.260471714" watchObservedRunningTime="2026-11-03 21:19:56.276962252 +0000 UTC m=+267.260958819"
Nov 03 21:20:00.625991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.625959    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rswc9"]
Nov 03 21:20:00.629249 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.629231    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:00.640130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.640107    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsm6\" (UniqueName: \"kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6\") pod \"authorino-57f7f8659f-rswc9\" (UID: \"7e719fac-37ad-4165-8cc4-4564c35fcca7\") " pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:00.686479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.686446    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t72bw\""
Nov 03 21:20:00.735916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.735878    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 21:20:00.735916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.735886    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 21:20:00.736791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.736770    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rswc9"]
Nov 03 21:20:00.741143 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.741115    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsm6\" (UniqueName: \"kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6\") pod \"authorino-57f7f8659f-rswc9\" (UID: \"7e719fac-37ad-4165-8cc4-4564c35fcca7\") " pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:00.801496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.801464    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsm6\" (UniqueName: \"kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6\") pod \"authorino-57f7f8659f-rswc9\" (UID: \"7e719fac-37ad-4165-8cc4-4564c35fcca7\") " pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:00.949185 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:00.949147    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:01.324996 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:01.324904    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rswc9"]
Nov 03 21:20:01.328678 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:20:01.328650    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e719fac_37ad_4165_8cc4_4564c35fcca7.slice/crio-bb6ede1d50e14abb8dcd495bcc12b1a95dba7167afbfe7b33c4396c6afdabf68 WatchSource:0}: Error finding container bb6ede1d50e14abb8dcd495bcc12b1a95dba7167afbfe7b33c4396c6afdabf68: Status 404 returned error can't find the container with id bb6ede1d50e14abb8dcd495bcc12b1a95dba7167afbfe7b33c4396c6afdabf68
Nov 03 21:20:02.186554 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:02.186515    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rswc9" event={"ID":"7e719fac-37ad-4165-8cc4-4564c35fcca7","Type":"ContainerStarted","Data":"bb6ede1d50e14abb8dcd495bcc12b1a95dba7167afbfe7b33c4396c6afdabf68"}
Nov 03 21:20:03.379342 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.379303    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:20:03.383743 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.383727    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.437403 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.437368    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 21:20:03.461990 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.461957    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c917408-568c-4cff-9395-3f6263463100-config-file\") pod \"limitador-limitador-7c9b87d574-qzfp9\" (UID: \"8c917408-568c-4cff-9395-3f6263463100\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.462155 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.462029    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfs2d\" (UniqueName: \"kubernetes.io/projected/8c917408-568c-4cff-9395-3f6263463100-kube-api-access-zfs2d\") pod \"limitador-limitador-7c9b87d574-qzfp9\" (UID: \"8c917408-568c-4cff-9395-3f6263463100\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.487871 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.487830    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:20:03.529775 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.529740    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvf9h\""
Nov 03 21:20:03.563046 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.563007    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfs2d\" (UniqueName: \"kubernetes.io/projected/8c917408-568c-4cff-9395-3f6263463100-kube-api-access-zfs2d\") pod \"limitador-limitador-7c9b87d574-qzfp9\" (UID: \"8c917408-568c-4cff-9395-3f6263463100\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.563239 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.563078    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c917408-568c-4cff-9395-3f6263463100-config-file\") pod \"limitador-limitador-7c9b87d574-qzfp9\" (UID: \"8c917408-568c-4cff-9395-3f6263463100\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.563799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.563778    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c917408-568c-4cff-9395-3f6263463100-config-file\") pod \"limitador-limitador-7c9b87d574-qzfp9\" (UID: \"8c917408-568c-4cff-9395-3f6263463100\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.619394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.619361    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfs2d\" (UniqueName: \"kubernetes.io/projected/8c917408-568c-4cff-9395-3f6263463100-kube-api-access-zfs2d\") pod \"limitador-limitador-7c9b87d574-qzfp9\" (UID: \"8c917408-568c-4cff-9395-3f6263463100\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:03.691500 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.691462    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:20:03.696654 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:03.696635    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:04.064053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:04.063978    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:20:04.067850 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:20:04.067816    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c917408_568c_4cff_9395_3f6263463100.slice/crio-43d323e2382953c4f1d2a5a5a7a8e972dfed0b046f13d77b565f2a6892c4f42d WatchSource:0}: Error finding container 43d323e2382953c4f1d2a5a5a7a8e972dfed0b046f13d77b565f2a6892c4f42d: Status 404 returned error can't find the container with id 43d323e2382953c4f1d2a5a5a7a8e972dfed0b046f13d77b565f2a6892c4f42d
Nov 03 21:20:04.196842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:04.196806    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9" event={"ID":"8c917408-568c-4cff-9395-3f6263463100","Type":"ContainerStarted","Data":"43d323e2382953c4f1d2a5a5a7a8e972dfed0b046f13d77b565f2a6892c4f42d"}
Nov 03 21:20:07.210283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:07.210243    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rswc9" event={"ID":"7e719fac-37ad-4165-8cc4-4564c35fcca7","Type":"ContainerStarted","Data":"edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6"}
Nov 03 21:20:07.321172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:07.321055    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rswc9" podStartSLOduration=1.599466492 podStartE2EDuration="7.321035211s" totalImagesPullingTime="5.721568719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:07.320100819 +0000 UTC m=+278.304097377" watchObservedRunningTime="2026-11-03 21:20:07.321035211 +0000 UTC m=+278.305031770"
Nov 03 21:20:09.220107 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:09.220071    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9" event={"ID":"8c917408-568c-4cff-9395-3f6263463100","Type":"ContainerStarted","Data":"b356e5ab44db116d2478476c707ba62dbe25f0aac201f7824073d3078d22969e"}
Nov 03 21:20:09.220571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:09.220156    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:09.332477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:09.332421    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9" podStartSLOduration=1.73446757 podStartE2EDuration="6.332403443s" totalImagesPullingTime="4.597935873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:09.331820094 +0000 UTC m=+280.315816653" watchObservedRunningTime="2026-11-03 21:20:09.332403443 +0000 UTC m=+280.316400003"
Nov 03 21:20:09.964715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:09.964685    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rswc9"]
Nov 03 21:20:09.964883 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:09.964855    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rswc9" podUID="7e719fac-37ad-4165-8cc4-4564c35fcca7" containerName="authorino" containerID="cri-o://edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6" gracePeriod=30
Nov 03 21:20:10.208024 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.207994    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:10.225602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.225531    5244 generic.go:356] "Generic (PLEG): container finished" podID="7e719fac-37ad-4165-8cc4-4564c35fcca7" containerID="edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6" exitCode=0
Nov 03 21:20:10.225602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.225577    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rswc9"
Nov 03 21:20:10.225992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.225621    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rswc9" event={"ID":"7e719fac-37ad-4165-8cc4-4564c35fcca7","Type":"ContainerDied","Data":"edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6"}
Nov 03 21:20:10.225992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.225660    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rswc9" event={"ID":"7e719fac-37ad-4165-8cc4-4564c35fcca7","Type":"ContainerDied","Data":"bb6ede1d50e14abb8dcd495bcc12b1a95dba7167afbfe7b33c4396c6afdabf68"}
Nov 03 21:20:10.225992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.225682    5244 scope.go:122] "RemoveContainer" containerID="edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6"
Nov 03 21:20:10.235436 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.235413    5244 scope.go:122] "RemoveContainer" containerID="edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6"
Nov 03 21:20:10.235719 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:20:10.235700    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6\": container with ID starting with edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6 not found: ID does not exist" containerID="edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6"
Nov 03 21:20:10.235778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.235729    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6"} err="failed to get container status \"edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6\": rpc error: code = NotFound desc = could not find container \"edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6\": container with ID starting with edfad98371cb568835c03bcb287bb4e2e7e035afb14754a10d54a643993950d6 not found: ID does not exist"
Nov 03 21:20:10.318716 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.318684    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6\" (UniqueName: \"kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6\") pod \"7e719fac-37ad-4165-8cc4-4564c35fcca7\" (UID: \"7e719fac-37ad-4165-8cc4-4564c35fcca7\") "
Nov 03 21:20:10.321426 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.321400    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6" pod "7e719fac-37ad-4165-8cc4-4564c35fcca7" (UID: "7e719fac-37ad-4165-8cc4-4564c35fcca7"). InnerVolumeSpecName "kube-api-access-8lsm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:20:10.419469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.419426    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsm6\" (UniqueName: \"kubernetes.io/projected/7e719fac-37ad-4165-8cc4-4564c35fcca7-kube-api-access-8lsm6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:20:10.645104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.642952    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rswc9"]
Nov 03 21:20:10.646460 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:10.646434    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rswc9"]
Nov 03 21:20:11.134069 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:11.134039    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e719fac-37ad-4165-8cc4-4564c35fcca7" path="/var/lib/kubelet/pods/7e719fac-37ad-4165-8cc4-4564c35fcca7/volumes"
Nov 03 21:20:20.226470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:20.226433    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"
Nov 03 21:20:22.435024 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:20:22.434985    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:20:29.099831 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.099793    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"]
Nov 03 21:20:29.100364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.100101    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7e719fac-37ad-4165-8cc4-4564c35fcca7" containerName="authorino"
Nov 03 21:20:29.100364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.100113    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e719fac-37ad-4165-8cc4-4564c35fcca7" containerName="authorino"
Nov 03 21:20:29.100364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.100184    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e719fac-37ad-4165-8cc4-4564c35fcca7" containerName="authorino"
Nov 03 21:20:29.105398 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.105369    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.158788 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.158759    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 21:20:29.159626 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.159600    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"]
Nov 03 21:20:29.175098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.175067    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grw5b\" (UniqueName: \"kubernetes.io/projected/db456a3d-1c18-4567-978f-0b653438b0d0-kube-api-access-grw5b\") pod \"postgres-7cdbf6dc44-msqdw\" (UID: \"db456a3d-1c18-4567-978f-0b653438b0d0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.175266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.175124    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db456a3d-1c18-4567-978f-0b653438b0d0-data\") pod \"postgres-7cdbf6dc44-msqdw\" (UID: \"db456a3d-1c18-4567-978f-0b653438b0d0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.207967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.207932    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:20:29.207967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.207934    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:20:29.207967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.207938    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-84b4c\""
Nov 03 21:20:29.275778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.275735    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5b\" (UniqueName: \"kubernetes.io/projected/db456a3d-1c18-4567-978f-0b653438b0d0-kube-api-access-grw5b\") pod \"postgres-7cdbf6dc44-msqdw\" (UID: \"db456a3d-1c18-4567-978f-0b653438b0d0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.275951 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.275788    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db456a3d-1c18-4567-978f-0b653438b0d0-data\") pod \"postgres-7cdbf6dc44-msqdw\" (UID: \"db456a3d-1c18-4567-978f-0b653438b0d0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.276117 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.276103    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/db456a3d-1c18-4567-978f-0b653438b0d0-data\") pod \"postgres-7cdbf6dc44-msqdw\" (UID: \"db456a3d-1c18-4567-978f-0b653438b0d0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.331994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.331955    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grw5b\" (UniqueName: \"kubernetes.io/projected/db456a3d-1c18-4567-978f-0b653438b0d0-kube-api-access-grw5b\") pod \"postgres-7cdbf6dc44-msqdw\" (UID: \"db456a3d-1c18-4567-978f-0b653438b0d0\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.417477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.417380    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:29.787111 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:29.787080    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"]
Nov 03 21:20:29.790379 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:20:29.790348    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb456a3d_1c18_4567_978f_0b653438b0d0.slice/crio-9b8676da6b4d2d99f2b4030b5b8c52f060b43d75da2faadca11c5b05c918d06f WatchSource:0}: Error finding container 9b8676da6b4d2d99f2b4030b5b8c52f060b43d75da2faadca11c5b05c918d06f: Status 404 returned error can't find the container with id 9b8676da6b4d2d99f2b4030b5b8c52f060b43d75da2faadca11c5b05c918d06f
Nov 03 21:20:30.305483 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:30.305447    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw" event={"ID":"db456a3d-1c18-4567-978f-0b653438b0d0","Type":"ContainerStarted","Data":"9b8676da6b4d2d99f2b4030b5b8c52f060b43d75da2faadca11c5b05c918d06f"}
Nov 03 21:20:37.338267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:37.338230    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw" event={"ID":"db456a3d-1c18-4567-978f-0b653438b0d0","Type":"ContainerStarted","Data":"2ef3365d88c375a52f548ab4fc79283da92f23f2d69ae8b490397edbcc6148d5"}
Nov 03 21:20:37.338723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:37.338346    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:37.450991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:37.450927    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw" podStartSLOduration=1.9421244130000002 podStartE2EDuration="8.450908511s" totalImagesPullingTime="6.508784098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:37.450270692 +0000 UTC m=+308.434267277" watchObservedRunningTime="2026-11-03 21:20:37.450908511 +0000 UTC m=+308.434905071"
Nov 03 21:20:43.372886 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:43.372855    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-msqdw"
Nov 03 21:20:50.975228 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:50.975190    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5767669ffc-kxhzw"]
Nov 03 21:20:50.977524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:50.977502    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.072038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.072005    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.072218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.072053    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprnp\" (UniqueName: \"kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.072218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.072076    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.079023 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.078996    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 21:20:51.079023 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.079020    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 21:20:51.079194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.079028    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 21:20:51.079194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.078995    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ln2c9\""
Nov 03 21:20:51.079194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.079066    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 21:20:51.079194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.079075    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 21:20:51.080181 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.080164    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5767669ffc-kxhzw"]
Nov 03 21:20:51.173060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.172978    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.173060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.173024    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprnp\" (UniqueName: \"kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.173060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.173059    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.175591 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.175573    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.175782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.175761    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.230115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.230079    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprnp\" (UniqueName: \"kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp\") pod \"maas-controller-5767669ffc-kxhzw\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") " pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.289090 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.289042    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:51.606103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.606068    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cdbc4ddd8-6drd4"]
Nov 03 21:20:51.608235 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.608218    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.669914 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.669869    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cdbc4ddd8-6drd4"]
Nov 03 21:20:51.707812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.707775    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5767669ffc-kxhzw"]
Nov 03 21:20:51.711258 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:20:51.711234    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191afc9f_8975_4503_bd66_268044f30d0e.slice/crio-5fdebfe8c11bbeee0426c823cd99d6388d948d43a530756a91d90cbb05bd10d9 WatchSource:0}: Error finding container 5fdebfe8c11bbeee0426c823cd99d6388d948d43a530756a91d90cbb05bd10d9: Status 404 returned error can't find the container with id 5fdebfe8c11bbeee0426c823cd99d6388d948d43a530756a91d90cbb05bd10d9
Nov 03 21:20:51.776807 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.776777    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2a36d9c-2c6a-475a-92d5-3caef091151b-webhook-cert\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.776986 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.776817    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc66m\" (UniqueName: \"kubernetes.io/projected/b2a36d9c-2c6a-475a-92d5-3caef091151b-kube-api-access-nc66m\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.776986 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.776843    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a36d9c-2c6a-475a-92d5-3caef091151b-metrics-certs\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.877998 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.877925    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2a36d9c-2c6a-475a-92d5-3caef091151b-webhook-cert\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.877998 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.877971    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc66m\" (UniqueName: \"kubernetes.io/projected/b2a36d9c-2c6a-475a-92d5-3caef091151b-kube-api-access-nc66m\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.878126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.878092    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a36d9c-2c6a-475a-92d5-3caef091151b-metrics-certs\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.880543 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.880516    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2a36d9c-2c6a-475a-92d5-3caef091151b-webhook-cert\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.880655 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.880551    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2a36d9c-2c6a-475a-92d5-3caef091151b-metrics-certs\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:51.934516 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:51.934483    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc66m\" (UniqueName: \"kubernetes.io/projected/b2a36d9c-2c6a-475a-92d5-3caef091151b-kube-api-access-nc66m\") pod \"maas-controller-6cdbc4ddd8-6drd4\" (UID: \"b2a36d9c-2c6a-475a-92d5-3caef091151b\") " pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:52.220051 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:52.219998    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:52.400869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:52.400829    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5767669ffc-kxhzw" event={"ID":"191afc9f-8975-4503-bd66-268044f30d0e","Type":"ContainerStarted","Data":"5fdebfe8c11bbeee0426c823cd99d6388d948d43a530756a91d90cbb05bd10d9"}
Nov 03 21:20:52.436707 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:20:52.436673    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:20:52.589880 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:52.589806    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cdbc4ddd8-6drd4"]
Nov 03 21:20:52.623236 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:20:52.623209    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a36d9c_2c6a_475a_92d5_3caef091151b.slice/crio-c2d1993c65d046f447b6ac8544d6ce662eb0bbe54d881e2955e40761fb152073 WatchSource:0}: Error finding container c2d1993c65d046f447b6ac8544d6ce662eb0bbe54d881e2955e40761fb152073: Status 404 returned error can't find the container with id c2d1993c65d046f447b6ac8544d6ce662eb0bbe54d881e2955e40761fb152073
Nov 03 21:20:53.406810 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:53.406764    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4" event={"ID":"b2a36d9c-2c6a-475a-92d5-3caef091151b","Type":"ContainerStarted","Data":"c2d1993c65d046f447b6ac8544d6ce662eb0bbe54d881e2955e40761fb152073"}
Nov 03 21:20:58.430993 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:58.430891    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4" event={"ID":"b2a36d9c-2c6a-475a-92d5-3caef091151b","Type":"ContainerStarted","Data":"2f6ae89cd14216ddf8dcf39334061dc87d68618112349100efb9f0a2f3c4ac72"}
Nov 03 21:20:58.431457 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:58.431013    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:20:58.432796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:58.432771    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5767669ffc-kxhzw" event={"ID":"191afc9f-8975-4503-bd66-268044f30d0e","Type":"ContainerStarted","Data":"65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7"}
Nov 03 21:20:58.432898 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:58.432822    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:20:58.565364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:58.565286    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4" podStartSLOduration=2.026282219 podStartE2EDuration="7.56526686s" totalImagesPullingTime="5.538984641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:58.564317523 +0000 UTC m=+329.548314085" watchObservedRunningTime="2026-11-03 21:20:58.56526686 +0000 UTC m=+329.549263419"
Nov 03 21:20:58.683031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:20:58.682968    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5767669ffc-kxhzw" podStartSLOduration=2.052924375 podStartE2EDuration="8.682951862s" totalImagesPullingTime="6.630027487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:58.682493822 +0000 UTC m=+329.666490403" watchObservedRunningTime="2026-11-03 21:20:58.682951862 +0000 UTC m=+329.666948425"
Nov 03 21:21:09.439707 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:09.439677    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:21:09.440254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:09.439753    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cdbc4ddd8-6drd4"
Nov 03 21:21:09.696022 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:09.695941    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5767669ffc-kxhzw"]
Nov 03 21:21:09.696178 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:09.696127    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5767669ffc-kxhzw" podUID="191afc9f-8975-4503-bd66-268044f30d0e" containerName="manager" containerID="cri-o://65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7" gracePeriod=30
Nov 03 21:21:09.935680 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:09.935656    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:21:10.007565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.007481    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert\") pod \"191afc9f-8975-4503-bd66-268044f30d0e\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") "
Nov 03 21:21:10.007565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.007531    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp\" (UniqueName: \"kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp\") pod \"191afc9f-8975-4503-bd66-268044f30d0e\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") "
Nov 03 21:21:10.007807 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.007601    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs\") pod \"191afc9f-8975-4503-bd66-268044f30d0e\" (UID: \"191afc9f-8975-4503-bd66-268044f30d0e\") "
Nov 03 21:21:10.009912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.009880    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs" pod "191afc9f-8975-4503-bd66-268044f30d0e" (UID: "191afc9f-8975-4503-bd66-268044f30d0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:10.010014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.009911    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert" pod "191afc9f-8975-4503-bd66-268044f30d0e" (UID: "191afc9f-8975-4503-bd66-268044f30d0e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:10.010014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.009948    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp" pod "191afc9f-8975-4503-bd66-268044f30d0e" (UID: "191afc9f-8975-4503-bd66-268044f30d0e"). InnerVolumeSpecName "kube-api-access-xprnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:10.108323 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.108286    5244 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-webhook-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:10.108323 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.108316    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprnp\" (UniqueName: \"kubernetes.io/projected/191afc9f-8975-4503-bd66-268044f30d0e-kube-api-access-xprnp\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:10.108323 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.108348    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/191afc9f-8975-4503-bd66-268044f30d0e-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:10.486620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.486583    5244 generic.go:356] "Generic (PLEG): container finished" podID="191afc9f-8975-4503-bd66-268044f30d0e" containerID="65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7" exitCode=0
Nov 03 21:21:10.487025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.486634    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5767669ffc-kxhzw"
Nov 03 21:21:10.487025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.486673    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5767669ffc-kxhzw" event={"ID":"191afc9f-8975-4503-bd66-268044f30d0e","Type":"ContainerDied","Data":"65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7"}
Nov 03 21:21:10.487025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.486730    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5767669ffc-kxhzw" event={"ID":"191afc9f-8975-4503-bd66-268044f30d0e","Type":"ContainerDied","Data":"5fdebfe8c11bbeee0426c823cd99d6388d948d43a530756a91d90cbb05bd10d9"}
Nov 03 21:21:10.487025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.486758    5244 scope.go:122] "RemoveContainer" containerID="65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7"
Nov 03 21:21:10.495927 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.495910    5244 scope.go:122] "RemoveContainer" containerID="65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7"
Nov 03 21:21:10.496202 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:21:10.496183    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7\": container with ID starting with 65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7 not found: ID does not exist" containerID="65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7"
Nov 03 21:21:10.496301 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.496209    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7"} err="failed to get container status \"65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7\": rpc error: code = NotFound desc = could not find container \"65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7\": container with ID starting with 65f434e78a72da8d1b1d5c5630a1036150ce7fad25e07b11f12fbaa84da64bb7 not found: ID does not exist"
Nov 03 21:21:10.649928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.649894    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5767669ffc-kxhzw"]
Nov 03 21:21:10.654976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:10.654950    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5767669ffc-kxhzw"]
Nov 03 21:21:11.133955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:11.133921    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191afc9f-8975-4503-bd66-268044f30d0e" path="/var/lib/kubelet/pods/191afc9f-8975-4503-bd66-268044f30d0e/volumes"
Nov 03 21:21:20.488602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.488563    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"]
Nov 03 21:21:20.489104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.489033    5244 cpu_manager.go:414] "containerMap: removing container" podUID="191afc9f-8975-4503-bd66-268044f30d0e" containerName="manager"
Nov 03 21:21:20.489104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.489051    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191afc9f-8975-4503-bd66-268044f30d0e" containerName="manager"
Nov 03 21:21:20.489214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.489120    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="191afc9f-8975-4503-bd66-268044f30d0e" containerName="manager"
Nov 03 21:21:20.498594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.498570    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593174 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593141    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ac9c37-d9fb-4e07-b242-f2edc1413f68-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593181    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7ks\" (UniqueName: \"kubernetes.io/projected/09ac9c37-d9fb-4e07-b242-f2edc1413f68-kube-api-access-nq7ks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593224    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593273    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593416    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593452    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.593536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.593471    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.597109 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.597076    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"]
Nov 03 21:21:20.694677 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694637    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.694677 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694675    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.694907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694702    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.694907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694839    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.694907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694882    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.695017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694920    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ac9c37-d9fb-4e07-b242-f2edc1413f68-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.695017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.694971    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7ks\" (UniqueName: \"kubernetes.io/projected/09ac9c37-d9fb-4e07-b242-f2edc1413f68-kube-api-access-nq7ks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.695537 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.695509    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.697316 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.697282    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ac9c37-d9fb-4e07-b242-f2edc1413f68-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.697427 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.697354    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.697744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.697719    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.697838 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.697789    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09ac9c37-d9fb-4e07-b242-f2edc1413f68-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.757075 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.756997    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ac9c37-d9fb-4e07-b242-f2edc1413f68-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.758929 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.758907    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7ks\" (UniqueName: \"kubernetes.io/projected/09ac9c37-d9fb-4e07-b242-f2edc1413f68-kube-api-access-nq7ks\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5nzpr\" (UID: \"09ac9c37-d9fb-4e07-b242-f2edc1413f68\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:20.810091 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:20.810056    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:21.231443 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.231396    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"]
Nov 03 21:21:21.235344 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:21:21.235304    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ac9c37_d9fb_4e07_b242_f2edc1413f68.slice/crio-e68738168dad186240373534f1bf9115d829209a6079779d2ef309247890a697 WatchSource:0}: Error finding container e68738168dad186240373534f1bf9115d829209a6079779d2ef309247890a697: Status 404 returned error can't find the container with id e68738168dad186240373534f1bf9115d829209a6079779d2ef309247890a697
Nov 03 21:21:21.237287 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.237251    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:21:21.237396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.237377    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:21:21.533574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.533476    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr" event={"ID":"09ac9c37-d9fb-4e07-b242-f2edc1413f68","Type":"ContainerStarted","Data":"1f7b6d495b02a93a43801327ef348278ac3d9cfd70794b7c31b0f4af0eacce9d"}
Nov 03 21:21:21.533574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.533535    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr" event={"ID":"09ac9c37-d9fb-4e07-b242-f2edc1413f68","Type":"ContainerStarted","Data":"e68738168dad186240373534f1bf9115d829209a6079779d2ef309247890a697"}
Nov 03 21:21:21.533574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.533568    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:21.649877 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:21.649822    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr" podStartSLOduration=1.6498093379999998 podStartE2EDuration="1.649809338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:21.648685445 +0000 UTC m=+352.632682025" watchObservedRunningTime="2026-11-03 21:21:21.649809338 +0000 UTC m=+352.633805897"
Nov 03 21:21:22.435630 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:21:22.435592    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:21:22.540289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:22.540258    5244 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5nzpr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:21:22.540683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:22.540310    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr" podUID="09ac9c37-d9fb-4e07-b242-f2edc1413f68" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:21:25.539056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:25.539020    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5nzpr"
Nov 03 21:21:25.756030 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:25.755992    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"]
Nov 03 21:21:25.756536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:25.756504    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerName="discovery" containerID="cri-o://d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3" gracePeriod=30
Nov 03 21:21:25.853171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:25.853141    5244 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2s5t5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.17:8080/ready\": dial tcp 10.133.0.17:8080: connect: connection refused" start-of-body=
Nov 03 21:21:25.853267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:25.853210    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.17:8080/ready\": dial tcp 10.133.0.17:8080: connect: connection refused"
Nov 03 21:21:26.010040 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.010012    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:21:26.042825 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.042798    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.042867    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.042892    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.042917    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.042941    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.043021    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.043113    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs\") pod \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\" (UID: \"12e7a1f8-ed50-4de4-b02d-7847c2f16352\") "
Nov 03 21:21:26.043513 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.043348    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:26.045368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.045302    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:26.045486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.045464    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "kube-api-access-sjhzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:26.045716 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.045698    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:26.045779 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.045715    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:26.045852 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.045834    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:26.045884 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.045847    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert" pod "12e7a1f8-ed50-4de4-b02d-7847c2f16352" (UID: "12e7a1f8-ed50-4de4-b02d-7847c2f16352"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:26.144421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144345    5244 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-dns-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.144421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144373    5244 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7a1f8-ed50-4de4-b02d-7847c2f16352-local-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.144421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144383    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhzl\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-kube-api-access-sjhzl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.144421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144398    5244 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-cacerts\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.144421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144411    5244 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-csr-ca-configmap\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.144682 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144431    5244 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-kubeconfig\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.144682 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.144438    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7a1f8-ed50-4de4-b02d-7847c2f16352-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:26.556224 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.556184    5244 generic.go:356] "Generic (PLEG): container finished" podID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerID="d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3" exitCode=0
Nov 03 21:21:26.556656 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.556266    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"
Nov 03 21:21:26.556656 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.556294    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" event={"ID":"12e7a1f8-ed50-4de4-b02d-7847c2f16352","Type":"ContainerDied","Data":"d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3"}
Nov 03 21:21:26.556656 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.556435    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5" event={"ID":"12e7a1f8-ed50-4de4-b02d-7847c2f16352","Type":"ContainerDied","Data":"7bff5c51058afca3fb6deaf465a58a57a6818dc7fa83fb10f08bce5237fff3c9"}
Nov 03 21:21:26.556656 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.556464    5244 scope.go:122] "RemoveContainer" containerID="d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3"
Nov 03 21:21:26.565600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.565570    5244 scope.go:122] "RemoveContainer" containerID="d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3"
Nov 03 21:21:26.565891 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:21:26.565867    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3\": container with ID starting with d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3 not found: ID does not exist" containerID="d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3"
Nov 03 21:21:26.565963 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.565905    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3"} err="failed to get container status \"d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3\": rpc error: code = NotFound desc = could not find container \"d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3\": container with ID starting with d427269d31f81ce111c9fccd8c643f58a38abd0548840c727ecb78dd8556aae3 not found: ID does not exist"
Nov 03 21:21:26.727376 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.727339    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"]
Nov 03 21:21:26.740629 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.740596    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2s5t5"]
Nov 03 21:21:26.811260 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.811176    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"]
Nov 03 21:21:26.811586 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.811572    5244 cpu_manager.go:414] "containerMap: removing container" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerName="discovery"
Nov 03 21:21:26.811637 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.811588    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerName="discovery"
Nov 03 21:21:26.811672 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.811653    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" containerName="discovery"
Nov 03 21:21:26.819217 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.819195    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:26.852414 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.852377    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-76bs6\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:26.852594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.852417    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ghv\" (UniqueName: \"kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv\") pod \"llmisvc-controller-manager-869d5dcc5d-76bs6\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:26.872173 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.872141    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"]
Nov 03 21:21:26.875446 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.875415    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5slnj\""
Nov 03 21:21:26.875608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.875448    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 21:21:26.953264 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.953228    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-76bs6\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:26.953486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.953272    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ghv\" (UniqueName: \"kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv\") pod \"llmisvc-controller-manager-869d5dcc5d-76bs6\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:26.955928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:26.955898    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-76bs6\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:27.009237 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:27.009205    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ghv\" (UniqueName: \"kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv\") pod \"llmisvc-controller-manager-869d5dcc5d-76bs6\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:27.131259 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:27.131161    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:27.134207 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:27.134181    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e7a1f8-ed50-4de4-b02d-7847c2f16352" path="/var/lib/kubelet/pods/12e7a1f8-ed50-4de4-b02d-7847c2f16352/volumes"
Nov 03 21:21:27.499342 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:27.499294    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"]
Nov 03 21:21:27.502653 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:21:27.502614    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod648e2a7e_fe59_471c_9631_55e494b4672f.slice/crio-cef33e2df5debf0b813cfc21a1da5dd024824cfbf90273392cf691a93554485b WatchSource:0}: Error finding container cef33e2df5debf0b813cfc21a1da5dd024824cfbf90273392cf691a93554485b: Status 404 returned error can't find the container with id cef33e2df5debf0b813cfc21a1da5dd024824cfbf90273392cf691a93554485b
Nov 03 21:21:27.504015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:27.503994    5244 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:21:27.563248 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:27.563214    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6" event={"ID":"648e2a7e-fe59-471c-9631-55e494b4672f","Type":"ContainerStarted","Data":"cef33e2df5debf0b813cfc21a1da5dd024824cfbf90273392cf691a93554485b"}
Nov 03 21:21:30.580128 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:30.580085    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6" event={"ID":"648e2a7e-fe59-471c-9631-55e494b4672f","Type":"ContainerStarted","Data":"3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc"}
Nov 03 21:21:30.580624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:30.580180    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:21:30.692270 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:30.692207    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6" podStartSLOduration=2.470200039 podStartE2EDuration="4.692185742s" totalImagesPullingTime="2.221985703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:30.69169081 +0000 UTC m=+361.675687400" watchObservedRunningTime="2026-11-03 21:21:30.692185742 +0000 UTC m=+361.676182302"
Nov 03 21:21:40.368283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:40.368096    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:21:49.363654 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.363618    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"]
Nov 03 21:21:49.366896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.366874    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.458872 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.458832    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7sw\" (UniqueName: \"kubernetes.io/projected/5168928d-67f0-459f-9658-4e280b7446a9-kube-api-access-gz7sw\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.459038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.458885    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5168928d-67f0-459f-9658-4e280b7446a9-metrics-certs\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.459038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.458963    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5168928d-67f0-459f-9658-4e280b7446a9-maas-api-tls\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.459038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.459033    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5168928d-67f0-459f-9658-4e280b7446a9-openshift-service-ca\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.468251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.468221    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 21:21:49.468422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.468309    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nbmd6\""
Nov 03 21:21:49.468476 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.468469    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 21:21:49.469471 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.469450    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"]
Nov 03 21:21:49.559894 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.559861    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7sw\" (UniqueName: \"kubernetes.io/projected/5168928d-67f0-459f-9658-4e280b7446a9-kube-api-access-gz7sw\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.560073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.559909    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5168928d-67f0-459f-9658-4e280b7446a9-metrics-certs\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.560073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.559940    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5168928d-67f0-459f-9658-4e280b7446a9-maas-api-tls\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.560073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.560008    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5168928d-67f0-459f-9658-4e280b7446a9-openshift-service-ca\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.560785 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.560761    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5168928d-67f0-459f-9658-4e280b7446a9-openshift-service-ca\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.562709 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.562686    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5168928d-67f0-459f-9658-4e280b7446a9-metrics-certs\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.562804 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.562743    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5168928d-67f0-459f-9658-4e280b7446a9-maas-api-tls\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.571002 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.570976    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bcdfdb8b7-wcv4t"]
Nov 03 21:21:49.574369 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.574350    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:49.617118 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.617044    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7sw\" (UniqueName: \"kubernetes.io/projected/5168928d-67f0-459f-9658-4e280b7446a9-kube-api-access-gz7sw\") pod \"maas-api-6c4b54ff46-bp45f\" (UID: \"5168928d-67f0-459f-9658-4e280b7446a9\") " pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.629454 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.629428    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t72bw\""
Nov 03 21:21:49.630570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.630547    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bcdfdb8b7-wcv4t"]
Nov 03 21:21:49.660590 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.660558    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz77h\" (UniqueName: \"kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h\") pod \"authorino-6bcdfdb8b7-wcv4t\" (UID: \"5b6c7b2d-f981-4add-899b-ffc2dd928379\") " pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:49.679100 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.679063    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:49.761794 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.761764    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz77h\" (UniqueName: \"kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h\") pod \"authorino-6bcdfdb8b7-wcv4t\" (UID: \"5b6c7b2d-f981-4add-899b-ffc2dd928379\") " pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:49.818091 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.818060    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz77h\" (UniqueName: \"kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h\") pod \"authorino-6bcdfdb8b7-wcv4t\" (UID: \"5b6c7b2d-f981-4add-899b-ffc2dd928379\") " pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:49.893259 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:49.893179    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:50.000198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:50.000166    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bcdfdb8b7-wcv4t"]
Nov 03 21:21:50.050694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:50.050617    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"]
Nov 03 21:21:50.052857 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:21:50.052828    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5168928d_67f0_459f_9658_4e280b7446a9.slice/crio-53d0676c0ceb3841eadb7ecd96cc77f77679ff792c4d71684908cad029921296 WatchSource:0}: Error finding container 53d0676c0ceb3841eadb7ecd96cc77f77679ff792c4d71684908cad029921296: Status 404 returned error can't find the container with id 53d0676c0ceb3841eadb7ecd96cc77f77679ff792c4d71684908cad029921296
Nov 03 21:21:50.263294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:50.263260    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bcdfdb8b7-wcv4t"]
Nov 03 21:21:50.266690 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:21:50.266666    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b6c7b2d_f981_4add_899b_ffc2dd928379.slice/crio-076b2ce1d6f7a0466e9e96ad361c0d85efe1d183fd1f4e6324b84d6657e313e3 WatchSource:0}: Error finding container 076b2ce1d6f7a0466e9e96ad361c0d85efe1d183fd1f4e6324b84d6657e313e3: Status 404 returned error can't find the container with id 076b2ce1d6f7a0466e9e96ad361c0d85efe1d183fd1f4e6324b84d6657e313e3
Nov 03 21:21:50.662720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:50.662627    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f" event={"ID":"5168928d-67f0-459f-9658-4e280b7446a9","Type":"ContainerStarted","Data":"53d0676c0ceb3841eadb7ecd96cc77f77679ff792c4d71684908cad029921296"}
Nov 03 21:21:50.663955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:50.663929    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t" event={"ID":"5b6c7b2d-f981-4add-899b-ffc2dd928379","Type":"ContainerStarted","Data":"076b2ce1d6f7a0466e9e96ad361c0d85efe1d183fd1f4e6324b84d6657e313e3"}
Nov 03 21:21:52.437185 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:21:52.437096    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:21:52.676185 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.676140    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t" event={"ID":"5b6c7b2d-f981-4add-899b-ffc2dd928379","Type":"ContainerStarted","Data":"88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167"}
Nov 03 21:21:52.676185 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.676175    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t" podUID="5b6c7b2d-f981-4add-899b-ffc2dd928379" containerName="authorino" containerID="cri-o://88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167" gracePeriod=30
Nov 03 21:21:52.678599 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.678572    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f" event={"ID":"5168928d-67f0-459f-9658-4e280b7446a9","Type":"ContainerStarted","Data":"afcef755c4282511e8259bc0db60f1a88ac5014686952d82a646ff45eb9a7b76"}
Nov 03 21:21:52.678723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.678714    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:21:52.786446 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.786391    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t" podStartSLOduration=2.430197876 podStartE2EDuration="3.786376253s" totalImagesPullingTime="1.356178377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:52.786364609 +0000 UTC m=+383.770361168" watchObservedRunningTime="2026-11-03 21:21:52.786376253 +0000 UTC m=+383.770372811"
Nov 03 21:21:52.900984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.900907    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f" podStartSLOduration=1.937295428 podStartE2EDuration="3.900888775s" totalImagesPullingTime="1.963593347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:52.899768444 +0000 UTC m=+383.883765042" watchObservedRunningTime="2026-11-03 21:21:52.900888775 +0000 UTC m=+383.884885335"
Nov 03 21:21:52.917119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:52.917095    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:53.091168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.091083    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h\" (UniqueName: \"kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h\") pod \"5b6c7b2d-f981-4add-899b-ffc2dd928379\" (UID: \"5b6c7b2d-f981-4add-899b-ffc2dd928379\") "
Nov 03 21:21:53.093476 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.093441    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h" pod "5b6c7b2d-f981-4add-899b-ffc2dd928379" (UID: "5b6c7b2d-f981-4add-899b-ffc2dd928379"). InnerVolumeSpecName "kube-api-access-lz77h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:53.192320 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.192286    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz77h\" (UniqueName: \"kubernetes.io/projected/5b6c7b2d-f981-4add-899b-ffc2dd928379-kube-api-access-lz77h\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:21:53.683812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.683774    5244 generic.go:356] "Generic (PLEG): container finished" podID="5b6c7b2d-f981-4add-899b-ffc2dd928379" containerID="88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167" exitCode=0
Nov 03 21:21:53.684264 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.683823    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t"
Nov 03 21:21:53.684264 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.683877    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t" event={"ID":"5b6c7b2d-f981-4add-899b-ffc2dd928379","Type":"ContainerDied","Data":"88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167"}
Nov 03 21:21:53.684264 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.683917    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bcdfdb8b7-wcv4t" event={"ID":"5b6c7b2d-f981-4add-899b-ffc2dd928379","Type":"ContainerDied","Data":"076b2ce1d6f7a0466e9e96ad361c0d85efe1d183fd1f4e6324b84d6657e313e3"}
Nov 03 21:21:53.684264 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.683934    5244 scope.go:122] "RemoveContainer" containerID="88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167"
Nov 03 21:21:53.692182 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.692163    5244 scope.go:122] "RemoveContainer" containerID="88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167"
Nov 03 21:21:53.692509 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:21:53.692481    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167\": container with ID starting with 88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167 not found: ID does not exist" containerID="88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167"
Nov 03 21:21:53.692563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.692521    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167"} err="failed to get container status \"88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167\": rpc error: code = NotFound desc = could not find container \"88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167\": container with ID starting with 88bd13fc2b289e58cf1ff5c8c9efb22c697c248679eca366f9dfb0be9d1b3167 not found: ID does not exist"
Nov 03 21:21:53.843814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.843783    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bcdfdb8b7-wcv4t"]
Nov 03 21:21:53.893204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:53.893163    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bcdfdb8b7-wcv4t"]
Nov 03 21:21:55.134195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:55.134151    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6c7b2d-f981-4add-899b-ffc2dd928379" path="/var/lib/kubelet/pods/5b6c7b2d-f981-4add-899b-ffc2dd928379/volumes"
Nov 03 21:21:58.690198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:21:58.690169    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c4b54ff46-bp45f"
Nov 03 21:22:01.586218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:01.586187    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:22:22.436009 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:22:22.435965    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:22:33.796685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.796638    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"]
Nov 03 21:22:33.797266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.797119    5244 cpu_manager.go:414] "containerMap: removing container" podUID="5b6c7b2d-f981-4add-899b-ffc2dd928379" containerName="authorino"
Nov 03 21:22:33.797266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.797140    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6c7b2d-f981-4add-899b-ffc2dd928379" containerName="authorino"
Nov 03 21:22:33.797266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.797260    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6c7b2d-f981-4add-899b-ffc2dd928379" containerName="authorino"
Nov 03 21:22:33.801532 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.801510    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:33.857101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.857067    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"]
Nov 03 21:22:33.934951 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.934920    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27601318-46e6-42fd-9ba7-a4f6d32538fc-cert\") pod \"llmisvc-controller-manager-6466c554d8-l7t5j\" (UID: \"27601318-46e6-42fd-9ba7-a4f6d32538fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:33.935126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:33.934985    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7v5\" (UniqueName: \"kubernetes.io/projected/27601318-46e6-42fd-9ba7-a4f6d32538fc-kube-api-access-9g7v5\") pod \"llmisvc-controller-manager-6466c554d8-l7t5j\" (UID: \"27601318-46e6-42fd-9ba7-a4f6d32538fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:34.036292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.036249    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27601318-46e6-42fd-9ba7-a4f6d32538fc-cert\") pod \"llmisvc-controller-manager-6466c554d8-l7t5j\" (UID: \"27601318-46e6-42fd-9ba7-a4f6d32538fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:34.036502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.036310    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7v5\" (UniqueName: \"kubernetes.io/projected/27601318-46e6-42fd-9ba7-a4f6d32538fc-kube-api-access-9g7v5\") pod \"llmisvc-controller-manager-6466c554d8-l7t5j\" (UID: \"27601318-46e6-42fd-9ba7-a4f6d32538fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:34.038845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.038819    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27601318-46e6-42fd-9ba7-a4f6d32538fc-cert\") pod \"llmisvc-controller-manager-6466c554d8-l7t5j\" (UID: \"27601318-46e6-42fd-9ba7-a4f6d32538fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:34.093318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.093232    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7v5\" (UniqueName: \"kubernetes.io/projected/27601318-46e6-42fd-9ba7-a4f6d32538fc-kube-api-access-9g7v5\") pod \"llmisvc-controller-manager-6466c554d8-l7t5j\" (UID: \"27601318-46e6-42fd-9ba7-a4f6d32538fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:34.114017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.113979    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:34.484852 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.484820    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"]
Nov 03 21:22:34.488112 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:22:34.488080    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27601318_46e6_42fd_9ba7_a4f6d32538fc.slice/crio-0c4b7f7ff86c1f626ed38ea7490f415b497bc57c85b4da3eef01642d4dd3b722 WatchSource:0}: Error finding container 0c4b7f7ff86c1f626ed38ea7490f415b497bc57c85b4da3eef01642d4dd3b722: Status 404 returned error can't find the container with id 0c4b7f7ff86c1f626ed38ea7490f415b497bc57c85b4da3eef01642d4dd3b722
Nov 03 21:22:34.853125 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:34.853043    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j" event={"ID":"27601318-46e6-42fd-9ba7-a4f6d32538fc","Type":"ContainerStarted","Data":"0c4b7f7ff86c1f626ed38ea7490f415b497bc57c85b4da3eef01642d4dd3b722"}
Nov 03 21:22:35.858514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:35.858480    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j" event={"ID":"27601318-46e6-42fd-9ba7-a4f6d32538fc","Type":"ContainerStarted","Data":"b323a9e8ca6a841d6bb65e341a148738500a03bef08d7f53124cc2a235804d93"}
Nov 03 21:22:35.858904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:35.858580    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:22:35.974903 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:22:35.974845    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j" podStartSLOduration=2.220327983 podStartE2EDuration="2.974829047s" totalImagesPullingTime="754.501064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:35.973983095 +0000 UTC m=+426.957979713" watchObservedRunningTime="2026-11-03 21:22:35.974829047 +0000 UTC m=+426.958825613"
Nov 03 21:22:52.437038 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:22:52.436995    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:23:06.865427 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:06.865350    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-l7t5j"
Nov 03 21:23:07.027798 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.027759    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"]
Nov 03 21:23:07.028062 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.028036    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6" podUID="648e2a7e-fe59-471c-9631-55e494b4672f" containerName="manager" containerID="cri-o://3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc" gracePeriod=30
Nov 03 21:23:07.270988 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.270962    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:23:07.317585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.317555    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert\" (UniqueName: \"kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert\") pod \"648e2a7e-fe59-471c-9631-55e494b4672f\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") "
Nov 03 21:23:07.317761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.317604    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv\" (UniqueName: \"kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv\") pod \"648e2a7e-fe59-471c-9631-55e494b4672f\" (UID: \"648e2a7e-fe59-471c-9631-55e494b4672f\") "
Nov 03 21:23:07.319761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.319731    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert" pod "648e2a7e-fe59-471c-9631-55e494b4672f" (UID: "648e2a7e-fe59-471c-9631-55e494b4672f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:23:07.319875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.319781    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv" pod "648e2a7e-fe59-471c-9631-55e494b4672f" (UID: "648e2a7e-fe59-471c-9631-55e494b4672f"). InnerVolumeSpecName "kube-api-access-v9ghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:23:07.419056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.418971    5244 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648e2a7e-fe59-471c-9631-55e494b4672f-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:23:07.419056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.419002    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ghv\" (UniqueName: \"kubernetes.io/projected/648e2a7e-fe59-471c-9631-55e494b4672f-kube-api-access-v9ghv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:23:07.989570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.989540    5244 generic.go:356] "Generic (PLEG): container finished" podID="648e2a7e-fe59-471c-9631-55e494b4672f" containerID="3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc" exitCode=0
Nov 03 21:23:07.990034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.989575    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6" event={"ID":"648e2a7e-fe59-471c-9631-55e494b4672f","Type":"ContainerDied","Data":"3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc"}
Nov 03 21:23:07.990034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.989592    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"
Nov 03 21:23:07.990034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.989644    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6" event={"ID":"648e2a7e-fe59-471c-9631-55e494b4672f","Type":"ContainerDied","Data":"cef33e2df5debf0b813cfc21a1da5dd024824cfbf90273392cf691a93554485b"}
Nov 03 21:23:07.990034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.989668    5244 scope.go:122] "RemoveContainer" containerID="3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc"
Nov 03 21:23:07.998364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.998317    5244 scope.go:122] "RemoveContainer" containerID="3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc"
Nov 03 21:23:07.998638 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:23:07.998619    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc\": container with ID starting with 3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc not found: ID does not exist" containerID="3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc"
Nov 03 21:23:07.998694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:07.998649    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc"} err="failed to get container status \"3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc\": rpc error: code = NotFound desc = could not find container \"3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc\": container with ID starting with 3b05b09eac5b6f092a1f187f377aaf2ab6756e91e2425e55103e763a965022fc not found: ID does not exist"
Nov 03 21:23:08.150824 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:08.150794    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"]
Nov 03 21:23:08.200136 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:08.200096    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-76bs6"]
Nov 03 21:23:09.134662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:23:09.134628    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648e2a7e-fe59-471c-9631-55e494b4672f" path="/var/lib/kubelet/pods/648e2a7e-fe59-471c-9631-55e494b4672f/volumes"
Nov 03 21:23:22.436421 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:23:22.436372    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:23:52.436153 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:23:52.436117    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:24:00.617680 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:00.617645    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:24:11.552108 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:11.552070    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:24:18.729855 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:18.729817    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:24:22.435422 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:24:22.435383    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:24:28.388843 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.388809    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"]
Nov 03 21:24:28.389349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.389313    5244 cpu_manager.go:414] "containerMap: removing container" podUID="648e2a7e-fe59-471c-9631-55e494b4672f" containerName="manager"
Nov 03 21:24:28.389428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.389349    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648e2a7e-fe59-471c-9631-55e494b4672f" containerName="manager"
Nov 03 21:24:28.389489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.389477    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="648e2a7e-fe59-471c-9631-55e494b4672f" containerName="manager"
Nov 03 21:24:28.392514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.392492    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.443966 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.443930    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mc2lq\""
Nov 03 21:24:28.530429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530394    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530437    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530646 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530525    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530646 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530582    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530646 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530601    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530742 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530694    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530775 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530742    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530775 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530768    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gqf\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.530836 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.530799    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.619528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.619492    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"]
Nov 03 21:24:28.631541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631512    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631691 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631548    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631691 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631566    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631685    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631743    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631828 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631767    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631828 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631787    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631921 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631834    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.631973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631939    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.632034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.631990    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gqf\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.632034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.632009    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.632142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.632054    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.632142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.632089    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.632622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.632589    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.634262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.634240    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.634863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.634839    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.688508 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.688478    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.688669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.688634    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gqf\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf\") pod \"maas-default-gateway-openshift-default-778b7b5c47-22pp7\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:28.709946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:28.709917    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:29.135292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:29.135213    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"]
Nov 03 21:24:29.176385 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:24:29.176322    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fc277d_6aae_4767_9706_c22e71ea9c02.slice/crio-b742c26d41645a102d7de45d172ab5422be80daf5760d0801f96572f09ace590 WatchSource:0}: Error finding container b742c26d41645a102d7de45d172ab5422be80daf5760d0801f96572f09ace590: Status 404 returned error can't find the container with id b742c26d41645a102d7de45d172ab5422be80daf5760d0801f96572f09ace590
Nov 03 21:24:29.324594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:29.324558    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" event={"ID":"28fc277d-6aae-4767-9706-c22e71ea9c02","Type":"ContainerStarted","Data":"b742c26d41645a102d7de45d172ab5422be80daf5760d0801f96572f09ace590"}
Nov 03 21:24:32.875349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:32.875295    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:24:32.875638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:32.875404    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:24:32.875638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:32.875450    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:24:33.372403 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:33.372306    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" event={"ID":"28fc277d-6aae-4767-9706-c22e71ea9c02","Type":"ContainerStarted","Data":"bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0"}
Nov 03 21:24:33.589901 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:33.589837    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podStartSLOduration=1.893851813 podStartE2EDuration="5.589659228s" totalImagesPullingTime="3.695807415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:33.589219103 +0000 UTC m=+544.573215662" watchObservedRunningTime="2026-11-03 21:24:33.589659228 +0000 UTC m=+544.573655787"
Nov 03 21:24:33.711179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:33.711141    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:33.712394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:33.712361    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:33.712538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:33.712417    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:33.851227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:33.851187    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:24:34.711227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:34.711180    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:34.711660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:34.711245    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:35.711087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:35.711049    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:35.711276 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:35.711103    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:36.711075 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:36.710998    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:36.711075 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:36.711057    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:37.710504 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:37.710468    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:37.710974 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:37.710526    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:38.710526 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:38.710487    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:38.711004 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:38.710547    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:39.710990 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:39.710948    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 21:24:39.711533 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:39.711015    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 21:24:41.751576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:41.751519    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:24:41.752041 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:41.751618    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:24:42.780126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:42.780080    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-22pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:24:42.780692 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:42.780142    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:24:42.792541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:42.792504    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:42.792921 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:42.792893    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:42.793761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:24:42.793746    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:24:52.438213 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:24:52.438176    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:25:22.436608 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:25:22.436568    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:25:47.055439 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:47.055405    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:25:51.182752 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.182715    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"]
Nov 03 21:25:51.186197 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.186178    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.238309 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.238276    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e7203b81-serving-cert\""
Nov 03 21:25:51.257744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.257707    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.257913 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.257764    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.257996 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.257970    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7kx6\" (UniqueName: \"kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.258038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.258024    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.288924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.288877    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"]
Nov 03 21:25:51.359698 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.359659    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.359896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.359749    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.359896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.359793    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.359896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.359829    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7kx6\" (UniqueName: \"kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.360061 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:25:51.359992    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-serving-cert: secret "maas-api-e2e-migrate-e7203b81-serving-cert" not found
Nov 03 21:25:51.360116 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:25:51.360076    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls podName:84fc353c-b3cb-4d78-8d77-8a4dd6c895b9 nodeName:}" failed. No retries permitted until 2026-11-03 21:25:51.860050269 +0000 UTC m=+622.844046809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls") pod "maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" (UID: "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9") : secret "maas-api-e2e-migrate-e7203b81-serving-cert" not found
Nov 03 21:25:51.360541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.360524    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.362249 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.362222    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.416695 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.416649    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7kx6\" (UniqueName: \"kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.866717 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.866683    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:51.869349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:51.869295    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\") pod \"maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:52.099265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:52.099231    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:52.435552 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:25:52.435517    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:25:52.468837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:52.468808    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"]
Nov 03 21:25:52.475125 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:25:52.475087    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fc353c_b3cb_4d78_8d77_8a4dd6c895b9.slice/crio-e3359f43f065089a773d3ad3815a5d57d8c231ac67d9014d5260f351af718f89 WatchSource:0}: Error finding container e3359f43f065089a773d3ad3815a5d57d8c231ac67d9014d5260f351af718f89: Status 404 returned error can't find the container with id e3359f43f065089a773d3ad3815a5d57d8c231ac67d9014d5260f351af718f89
Nov 03 21:25:52.726540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:52.726435    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" event={"ID":"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9","Type":"ContainerStarted","Data":"e3359f43f065089a773d3ad3815a5d57d8c231ac67d9014d5260f351af718f89"}
Nov 03 21:25:53.732097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:53.732061    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" event={"ID":"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9","Type":"ContainerStarted","Data":"a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60"}
Nov 03 21:25:53.732645 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:53.732185    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:25:53.845965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:53.845911    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" podStartSLOduration=1.997926283 podStartE2EDuration="2.845896641s" totalImagesPullingTime="847.970358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:53.845299292 +0000 UTC m=+624.829295854" watchObservedRunningTime="2026-11-03 21:25:53.845896641 +0000 UTC m=+624.829893199"
Nov 03 21:25:59.741790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:25:59.741761    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:26:00.585450 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.585417    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"]
Nov 03 21:26:00.589008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.588991    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.640577 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.640543    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-daf21c29-serving-cert\""
Nov 03 21:26:00.691090 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.691056    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"]
Nov 03 21:26:00.745463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.745433    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.745463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.745466    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.745901 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.745495    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.745901 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.745514    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kg8\" (UniqueName: \"kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.846578 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.846479    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.846578 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.846530    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.846578 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.846566    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.846862 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.846590    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kg8\" (UniqueName: \"kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.846862 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:26:00.846631    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-serving-cert: secret "maas-api-e2e-derive-daf21c29-serving-cert" not found
Nov 03 21:26:00.846862 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:26:00.846710    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls podName:3c866b96-5ccd-4150-ab65-aee3549b0526 nodeName:}" failed. No retries permitted until 2026-11-03 21:26:01.346688409 +0000 UTC m=+632.330684947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls") pod "maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" (UID: "3c866b96-5ccd-4150-ab65-aee3549b0526") : secret "maas-api-e2e-derive-daf21c29-serving-cert" not found
Nov 03 21:26:00.847478 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.847403    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.849192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.849174    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:00.903420 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:00.903388    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kg8\" (UniqueName: \"kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:01.350975 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:01.350933    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:01.353590 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:01.353563    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\") pod \"maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:01.502788 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:01.502754    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:01.875024 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:01.874996    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"]
Nov 03 21:26:01.878440 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:26:01.878414    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c866b96_5ccd_4150_ab65_aee3549b0526.slice/crio-eea52bb84ade9644fc813f43ed96051630c382b2ee8fe928275d91b8eed4bbe0 WatchSource:0}: Error finding container eea52bb84ade9644fc813f43ed96051630c382b2ee8fe928275d91b8eed4bbe0: Status 404 returned error can't find the container with id eea52bb84ade9644fc813f43ed96051630c382b2ee8fe928275d91b8eed4bbe0
Nov 03 21:26:02.776963 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:02.775481    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" event={"ID":"3c866b96-5ccd-4150-ab65-aee3549b0526","Type":"ContainerStarted","Data":"eea52bb84ade9644fc813f43ed96051630c382b2ee8fe928275d91b8eed4bbe0"}
Nov 03 21:26:03.781522 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:03.781487    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" event={"ID":"3c866b96-5ccd-4150-ab65-aee3549b0526","Type":"ContainerStarted","Data":"7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257"}
Nov 03 21:26:03.781942 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:03.781615    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:03.894666 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:03.894605    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" podStartSLOduration=3.043354433 podStartE2EDuration="3.89459045s" totalImagesPullingTime="851.236017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:03.893739943 +0000 UTC m=+634.877736502" watchObservedRunningTime="2026-11-03 21:26:03.89459045 +0000 UTC m=+634.878587009"
Nov 03 21:26:06.634423 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:06.634313    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:26:09.791104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:09.791074    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:26:22.437947 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:26:22.437891    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:26:41.531048 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:41.531005    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:26:42.204169 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.204134    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"]
Nov 03 21:26:42.204398 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.204374    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" podUID="84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" containerName="maas-api" containerID="cri-o://a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60" gracePeriod=30
Nov 03 21:26:42.459375 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.459299    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:26:42.541260 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.541212    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca\") pod \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") "
Nov 03 21:26:42.541819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.541262    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6\" (UniqueName: \"kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6\") pod \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") "
Nov 03 21:26:42.541819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.541316    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs\") pod \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") "
Nov 03 21:26:42.541819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.541494    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\") pod \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\" (UID: \"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9\") "
Nov 03 21:26:42.541819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.541721    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca" pod "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" (UID: "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:26:42.541997 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.541829    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:42.543812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.543784    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs" pod "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" (UID: "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:42.543926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.543818    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls" pod "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" (UID: "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:26:42.543926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.543861    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6" pod "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" (UID: "84fc353c-b3cb-4d78-8d77-8a4dd6c895b9"). InnerVolumeSpecName "kube-api-access-r7kx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:26:42.642615 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.642575    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7kx6\" (UniqueName: \"kubernetes.io/projected/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-kube-api-access-r7kx6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:42.642615 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.642616    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:42.643019 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.642629    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:26:42.958092 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.958058    5244 generic.go:356] "Generic (PLEG): container finished" podID="84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" containerID="a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60" exitCode=0
Nov 03 21:26:42.958315 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.958124    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"
Nov 03 21:26:42.958315 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.958137    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" event={"ID":"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9","Type":"ContainerDied","Data":"a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60"}
Nov 03 21:26:42.958315 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.958178    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl" event={"ID":"84fc353c-b3cb-4d78-8d77-8a4dd6c895b9","Type":"ContainerDied","Data":"e3359f43f065089a773d3ad3815a5d57d8c231ac67d9014d5260f351af718f89"}
Nov 03 21:26:42.958315 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.958198    5244 scope.go:122] "RemoveContainer" containerID="a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60"
Nov 03 21:26:42.967489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.967466    5244 scope.go:122] "RemoveContainer" containerID="a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60"
Nov 03 21:26:42.967845 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:26:42.967820    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60\": container with ID starting with a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60 not found: ID does not exist" containerID="a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60"
Nov 03 21:26:42.967949 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:42.967852    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60"} err="failed to get container status \"a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60\": rpc error: code = NotFound desc = could not find container \"a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60\": container with ID starting with a1c0d00f60b08d8403732ba8efb947ed77488416333b31c96994dbb6c4651f60 not found: ID does not exist"
Nov 03 21:26:43.121795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:43.121754    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"]
Nov 03 21:26:43.125222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:43.125200    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e7203b81-76ddf58ff7-bf8hl"]
Nov 03 21:26:43.134201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:43.134173    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" path="/var/lib/kubelet/pods/84fc353c-b3cb-4d78-8d77-8a4dd6c895b9/volumes"
Nov 03 21:26:51.210452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:26:51.210410    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:26:52.436437 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:26:52.436402    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:27:08.450318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:08.450274    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:27:15.039277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.039238    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"]
Nov 03 21:27:15.039846 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.039484    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" podUID="3c866b96-5ccd-4150-ab65-aee3549b0526" containerName="maas-api" containerID="cri-o://7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257" gracePeriod=30
Nov 03 21:27:15.290856 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.290790    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:27:15.351773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.351738    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8\" (UniqueName: \"kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8\") pod \"3c866b96-5ccd-4150-ab65-aee3549b0526\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") "
Nov 03 21:27:15.351773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.351775    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs\") pod \"3c866b96-5ccd-4150-ab65-aee3549b0526\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") "
Nov 03 21:27:15.351983 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.351824    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca\") pod \"3c866b96-5ccd-4150-ab65-aee3549b0526\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") "
Nov 03 21:27:15.351983 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.351939    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\") pod \"3c866b96-5ccd-4150-ab65-aee3549b0526\" (UID: \"3c866b96-5ccd-4150-ab65-aee3549b0526\") "
Nov 03 21:27:15.352210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.352189    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca" pod "3c866b96-5ccd-4150-ab65-aee3549b0526" (UID: "3c866b96-5ccd-4150-ab65-aee3549b0526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:27:15.354192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.354163    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls" pod "3c866b96-5ccd-4150-ab65-aee3549b0526" (UID: "3c866b96-5ccd-4150-ab65-aee3549b0526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:15.354303 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.354202    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs" pod "3c866b96-5ccd-4150-ab65-aee3549b0526" (UID: "3c866b96-5ccd-4150-ab65-aee3549b0526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:27:15.354303 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.354242    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8" pod "3c866b96-5ccd-4150-ab65-aee3549b0526" (UID: "3c866b96-5ccd-4150-ab65-aee3549b0526"). InnerVolumeSpecName "kube-api-access-h8kg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:27:15.453057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.453015    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kg8\" (UniqueName: \"kubernetes.io/projected/3c866b96-5ccd-4150-ab65-aee3549b0526-kube-api-access-h8kg8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:15.453057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.453054    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:15.453057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.453067    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c866b96-5ccd-4150-ab65-aee3549b0526-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:15.453346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:15.453080    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c866b96-5ccd-4150-ab65-aee3549b0526-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:27:16.124687 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.124654    5244 generic.go:356] "Generic (PLEG): container finished" podID="3c866b96-5ccd-4150-ab65-aee3549b0526" containerID="7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257" exitCode=0
Nov 03 21:27:16.125119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.124699    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"
Nov 03 21:27:16.125119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.124746    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" event={"ID":"3c866b96-5ccd-4150-ab65-aee3549b0526","Type":"ContainerDied","Data":"7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257"}
Nov 03 21:27:16.125119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.124805    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l" event={"ID":"3c866b96-5ccd-4150-ab65-aee3549b0526","Type":"ContainerDied","Data":"eea52bb84ade9644fc813f43ed96051630c382b2ee8fe928275d91b8eed4bbe0"}
Nov 03 21:27:16.125119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.124830    5244 scope.go:122] "RemoveContainer" containerID="7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257"
Nov 03 21:27:16.134884 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.134867    5244 scope.go:122] "RemoveContainer" containerID="7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257"
Nov 03 21:27:16.135134 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:27:16.135112    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257\": container with ID starting with 7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257 not found: ID does not exist" containerID="7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257"
Nov 03 21:27:16.135233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.135143    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257"} err="failed to get container status \"7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257\": rpc error: code = NotFound desc = could not find container \"7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257\": container with ID starting with 7fdcef6fa81059516e30e4a599749c5a1de789c63039457fcfe5c387f9383257 not found: ID does not exist"
Nov 03 21:27:16.286076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.286044    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"]
Nov 03 21:27:16.335552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:16.335515    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-daf21c29-5cb58fb5b9-dl46l"]
Nov 03 21:27:17.134512 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:17.134473    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c866b96-5ccd-4150-ab65-aee3549b0526" path="/var/lib/kubelet/pods/3c866b96-5ccd-4150-ab65-aee3549b0526/volumes"
Nov 03 21:27:18.755784 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:18.755749    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:27:22.436708 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:27:22.436670    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:27:26.899152 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899115    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"]
Nov 03 21:27:26.899611 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899515    5244 cpu_manager.go:414] "containerMap: removing container" podUID="3c866b96-5ccd-4150-ab65-aee3549b0526" containerName="maas-api"
Nov 03 21:27:26.899611 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899530    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c866b96-5ccd-4150-ab65-aee3549b0526" containerName="maas-api"
Nov 03 21:27:26.899611 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899563    5244 cpu_manager.go:414] "containerMap: removing container" podUID="84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" containerName="maas-api"
Nov 03 21:27:26.899611 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899568    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" containerName="maas-api"
Nov 03 21:27:26.899748 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899621    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fc353c-b3cb-4d78-8d77-8a4dd6c895b9" containerName="maas-api"
Nov 03 21:27:26.899748 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.899629    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c866b96-5ccd-4150-ab65-aee3549b0526" containerName="maas-api"
Nov 03 21:27:26.903797 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.903780    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.968752 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.968714    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.968752 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.968757    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.969014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.968825    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.969014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.968841    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.969014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.968858    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqgv\" (UniqueName: \"kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.969014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.968969    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:26.981887 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:26.981854    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"]
Nov 03 21:27:27.005036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.004998    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"default-dockercfg-xz5mb\""
Nov 03 21:27:27.005036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.004998    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"openshift-service-ca.crt\""
Nov 03 21:27:27.005036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.005033    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"simulator-w2-d71a74-kserve-self-signed-certs\""
Nov 03 21:27:27.005265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.005007    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"kube-root-ca.crt\""
Nov 03 21:27:27.069506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069480    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.069506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069510    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.069767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069537    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.069767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069552    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.069767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069571    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqgv\" (UniqueName: \"kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.069767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069593    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.069985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.069961    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.070018    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.070078 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.070036    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.072055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.072039    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.072215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.072198    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.126510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.126471    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqgv\" (UniqueName: \"kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv\") pod \"simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") " pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.225594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.225555    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:27.599316 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.599233    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"]
Nov 03 21:27:27.603130 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:27:27.603102    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70773fa8_3c22_4f5f_8086_cc786e65c043.slice/crio-3f3f399b2fdd648503ce438aa107b99733ffc8d5bc515812d3b1d394c927dd6a WatchSource:0}: Error finding container 3f3f399b2fdd648503ce438aa107b99733ffc8d5bc515812d3b1d394c927dd6a: Status 404 returned error can't find the container with id 3f3f399b2fdd648503ce438aa107b99733ffc8d5bc515812d3b1d394c927dd6a
Nov 03 21:27:27.604961 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:27.604946    5244 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:27:28.191662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:28.191629    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" event={"ID":"70773fa8-3c22-4f5f-8086-cc786e65c043","Type":"ContainerStarted","Data":"3f3f399b2fdd648503ce438aa107b99733ffc8d5bc515812d3b1d394c927dd6a"}
Nov 03 21:27:30.204231 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:30.204191    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" event={"ID":"70773fa8-3c22-4f5f-8086-cc786e65c043","Type":"ContainerStarted","Data":"20bb461da74e86996bb5988499c52b6bd1fa9fe6c596e214e4e1a018e35226f5"}
Nov 03 21:27:30.226627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:30.226589    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:30.320552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:30.320495    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" podStartSLOduration=2.303604425 podStartE2EDuration="4.320475469s" totalImagesPullingTime="2.016871044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:30.318788827 +0000 UTC m=+721.302785437" watchObservedRunningTime="2026-11-03 21:27:30.320475469 +0000 UTC m=+721.304472028"
Nov 03 21:27:35.220530 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:35.220484    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:27:37.226590 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:37.226509    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:37.239087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:37.239061    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:37.250703 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:37.250670    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:27:51.901818 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:51.901785    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"]
Nov 03 21:27:51.905206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:51.905190    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:51.957185 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:51.957152    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-f463c9-serving-cert\""
Nov 03 21:27:52.004747 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.004709    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.004931 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.004840    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.004931 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.004869    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjzm\" (UniqueName: \"kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.004931 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.004911    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.008024 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.008001    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"]
Nov 03 21:27:52.106057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.106026    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.106229 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.106076    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.106229 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.106121    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.106229 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:27:52.106202    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-serving-cert: secret "maas-api-e2e-worker-w1-f463c9-serving-cert" not found
Nov 03 21:27:52.106410 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.106252    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjzm\" (UniqueName: \"kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.106410 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:27:52.106277    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls podName:dd9bb00a-9e87-4091-957e-a09f10c147e0 nodeName:}" failed. No retries permitted until 2026-11-03 21:27:52.606255061 +0000 UTC m=+743.590251616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls") pod "maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" (UID: "dd9bb00a-9e87-4091-957e-a09f10c147e0") : secret "maas-api-e2e-worker-w1-f463c9-serving-cert" not found
Nov 03 21:27:52.106930 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.106899    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.108896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.108875    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.163280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.163197    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjzm\" (UniqueName: \"kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.438107 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:27:52.438068    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:27:52.610667 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.610624    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.613362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.613317    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:52.817440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:52.817354    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:53.191053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:53.191022    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"]
Nov 03 21:27:53.194377 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:27:53.194346    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9bb00a_9e87_4091_957e_a09f10c147e0.slice/crio-7c533164224de795669e4562c6c4a82c06f5f6c1fe2e51c5671e6deb384ae310 WatchSource:0}: Error finding container 7c533164224de795669e4562c6c4a82c06f5f6c1fe2e51c5671e6deb384ae310: Status 404 returned error can't find the container with id 7c533164224de795669e4562c6c4a82c06f5f6c1fe2e51c5671e6deb384ae310
Nov 03 21:27:53.310196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:53.310158    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" event={"ID":"dd9bb00a-9e87-4091-957e-a09f10c147e0","Type":"ContainerStarted","Data":"7c533164224de795669e4562c6c4a82c06f5f6c1fe2e51c5671e6deb384ae310"}
Nov 03 21:27:54.317051 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.316963    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" event={"ID":"dd9bb00a-9e87-4091-957e-a09f10c147e0","Type":"ContainerStarted","Data":"4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d"}
Nov 03 21:27:54.317477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.317061    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:27:54.432703 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.432636    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" podStartSLOduration=2.567051841 podStartE2EDuration="3.432616697s" totalImagesPullingTime="865.564856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:54.432611262 +0000 UTC m=+745.416607840" watchObservedRunningTime="2026-11-03 21:27:54.432616697 +0000 UTC m=+745.416613256"
Nov 03 21:27:54.746799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.746767    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"]
Nov 03 21:27:54.750084 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.750068    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.802433 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.802401    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"kube-root-ca.crt\""
Nov 03 21:27:54.829060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.829017    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"default-dockercfg-qfhdp\""
Nov 03 21:27:54.829060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.829017    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"openshift-service-ca.crt\""
Nov 03 21:27:54.839429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.839404    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.839530 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.839457    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.839530 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.839507    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.839625 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.839530    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.839625 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.839587    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjpv\" (UniqueName: \"kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.839625 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.839606    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.851622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.851595    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"simulator-w0-7ee809-kserve-self-signed-certs\""
Nov 03 21:27:54.852744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.852725    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"]
Nov 03 21:27:54.940266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940223    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940271    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940357    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940376    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940483    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940703 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940579    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjpv\" (UniqueName: \"kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940801 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940772    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940805    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.940932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.940885    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.942748 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.942724    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.943022 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.943004    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:54.998088 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:54.998002    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjpv\" (UniqueName: \"kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv\") pod \"simulator-w0-7ee809-kserve-6b6d545c85-dd4lj\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") " pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:55.061823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:55.061785    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:55.436616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:55.436584    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"]
Nov 03 21:27:56.326816 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:56.326780    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" event={"ID":"4fd6e820-8ece-4b96-b89b-82643f7dd79f","Type":"ContainerStarted","Data":"cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4"}
Nov 03 21:27:56.327021 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:56.326822    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" event={"ID":"4fd6e820-8ece-4b96-b89b-82643f7dd79f","Type":"ContainerStarted","Data":"a123cc1fead60b1af999a1fcdd1ab345a3aab10bf6776078ab0ea13241e8af47"}
Nov 03 21:27:56.440428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:56.440365    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" podStartSLOduration=2.440348218 podStartE2EDuration="2.440348218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:56.439684437 +0000 UTC m=+747.423681020" watchObservedRunningTime="2026-11-03 21:27:56.440348218 +0000 UTC m=+747.424344783"
Nov 03 21:27:57.062004 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:57.061959    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:27:58.203803 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.203762    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"]
Nov 03 21:27:58.207480 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.207456    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.274506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.274473    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.274675 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.274514    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.274675 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.274535    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xklg\" (UniqueName: \"kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.274783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.274713    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.274862 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.274844    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.274922 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.274890    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.308631 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.308600    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"openshift-service-ca.crt\""
Nov 03 21:27:58.308631 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.308622    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"default-dockercfg-cmzvh\""
Nov 03 21:27:58.308878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.308651    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"simulator-w3-da5f0b-kserve-self-signed-certs\""
Nov 03 21:27:58.308878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.308747    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-da5f0b\"/\"kube-root-ca.crt\""
Nov 03 21:27:58.310068 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.310048    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"]
Nov 03 21:27:58.376214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376184    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376224    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376260    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376280    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376303    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xklg\" (UniqueName: \"kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376369    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376699 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376677    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376828 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376805    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.376878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.376834    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.378549 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.378530    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.378839 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.378819    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.433100 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.433068    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xklg\" (UniqueName: \"kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg\") pod \"simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") " pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.519580 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.519478    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:27:58.899373 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:58.899170    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"]
Nov 03 21:27:58.901971 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:27:58.901945    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9ed677e_1d69_432d_9d1b_2b6b7c9216f3.slice/crio-c301a91eb4f35bf8df68186d31e79e35e07eea87e47fbd9f7bdedb35ead28652 WatchSource:0}: Error finding container c301a91eb4f35bf8df68186d31e79e35e07eea87e47fbd9f7bdedb35ead28652: Status 404 returned error can't find the container with id c301a91eb4f35bf8df68186d31e79e35e07eea87e47fbd9f7bdedb35ead28652
Nov 03 21:27:59.342493 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:59.342450    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" event={"ID":"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3","Type":"ContainerStarted","Data":"2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033"}
Nov 03 21:27:59.342493 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:59.342490    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" event={"ID":"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3","Type":"ContainerStarted","Data":"c301a91eb4f35bf8df68186d31e79e35e07eea87e47fbd9f7bdedb35ead28652"}
Nov 03 21:27:59.457605 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:59.457553    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podStartSLOduration=1.4575371719999999 podStartE2EDuration="1.457537172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:59.456374377 +0000 UTC m=+750.440370955" watchObservedRunningTime="2026-11-03 21:27:59.457537172 +0000 UTC m=+750.441533773"
Nov 03 21:27:59.519907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:27:59.519872    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:28:00.326672 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:00.326645    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 21:28:01.076084 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:01.076050    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:28:05.062079 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:05.062042    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:28:05.074669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:05.074642    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:28:05.381784 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:05.381706    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:28:08.519697 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:08.519664    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:28:08.532514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:08.532487    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:28:09.400663 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:09.400635    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:28:22.437606 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:28:22.437563    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:28:27.353911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.353878    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"]
Nov 03 21:28:27.357583 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.357562    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.449875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.449844    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.449875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.449891    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.450111 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.449916    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.450111 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.450002    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.450111 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.450031    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzt6\" (UniqueName: \"kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.450111 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.450060    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.458791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.458762    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"premium-w2-d71a74-kserve-self-signed-certs\""
Nov 03 21:28:27.460119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.460095    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"]
Nov 03 21:28:27.550716 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.550676    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.550907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.550736    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.550907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.550771    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.550907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.550795    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.550907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.550821    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzt6\" (UniqueName: \"kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.550907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.550852    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.551284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.551265    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.551368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.551296    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.551368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.551299    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.553274 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.553252    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.553600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.553583    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.607468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.607378    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzt6\" (UniqueName: \"kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6\") pod \"premium-w2-d71a74-kserve-7cd79dd77b-6l9bj\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") " pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:27.670391 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:27.670344    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:28.053681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:28.052901    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"]
Nov 03 21:28:28.477585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:28.477550    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" event={"ID":"136a8937-a0c9-4e97-bf99-f4e708359238","Type":"ContainerStarted","Data":"238123f07649f7d08ff0efcb54f6cf8aeaf9342cf57ac95f9edfd01183045a8d"}
Nov 03 21:28:28.477585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:28.477592    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" event={"ID":"136a8937-a0c9-4e97-bf99-f4e708359238","Type":"ContainerStarted","Data":"83141846d63cb06bfbf6a91e9e23df43e4463407ccef4c56cfff5fadc8c167d1"}
Nov 03 21:28:28.591824 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:28.591769    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" podStartSLOduration=1.591755085 podStartE2EDuration="1.591755085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:28.590732092 +0000 UTC m=+779.574728674" watchObservedRunningTime="2026-11-03 21:28:28.591755085 +0000 UTC m=+779.575751643"
Nov 03 21:28:28.671126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:28.671092    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:35.206556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:35.206513    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:28:37.671006 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:37.670968    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:37.683346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:37.683298    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:38.532956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:38.532915    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:28:52.435533 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:28:52.435489    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:28:53.960298 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:28:53.960263    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:29:11.401472 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.401437    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"]
Nov 03 21:29:11.404814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.404791    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.456589 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.456555    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"openshift-service-ca.crt\""
Nov 03 21:29:11.467169 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.467137    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.467391 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.467181    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntfc2\" (UniqueName: \"kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.467391 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.467202    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.467391 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.467355    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.467555 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.467505    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.467555 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.467545    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.506081 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.506044    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"kube-root-ca.crt\""
Nov 03 21:29:11.506081 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.506075    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"default-dockercfg-vq2z2\""
Nov 03 21:29:11.506309 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.506044    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"simulator-w1-f463c9-kserve-self-signed-certs\""
Nov 03 21:29:11.507476 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.507458    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"]
Nov 03 21:29:11.568489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568447    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.568681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568507    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.568681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568618    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.568681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568643    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.568912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568701    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.568912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568752    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntfc2\" (UniqueName: \"kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.569059 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.568961    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.569059 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.569033    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.569256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.569205    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.571939 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.571914    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.573018 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.572991    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.625102 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.625065    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntfc2\" (UniqueName: \"kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2\") pod \"simulator-w1-f463c9-kserve-849f45855-47f4f\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") " pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:11.716947 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:11.716904    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:12.095845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:12.095760    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"]
Nov 03 21:29:12.099256 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:29:12.099230    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd568b2bb_57fb_46c1_b824_7e249a7cfb49.slice/crio-aabf59e5647bb0163e9388e2aafc26c039209ed298482ed9a09561267d724d54 WatchSource:0}: Error finding container aabf59e5647bb0163e9388e2aafc26c039209ed298482ed9a09561267d724d54: Status 404 returned error can't find the container with id aabf59e5647bb0163e9388e2aafc26c039209ed298482ed9a09561267d724d54
Nov 03 21:29:12.672159 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:12.672128    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" event={"ID":"d568b2bb-57fb-46c1-b824-7e249a7cfb49","Type":"ContainerStarted","Data":"41e31d4e5604901fb0cd9f04a132e539db6c60c92fb9b996b4cd5377ade0fe43"}
Nov 03 21:29:12.672159 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:12.672168    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" event={"ID":"d568b2bb-57fb-46c1-b824-7e249a7cfb49","Type":"ContainerStarted","Data":"aabf59e5647bb0163e9388e2aafc26c039209ed298482ed9a09561267d724d54"}
Nov 03 21:29:12.717565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:12.717539    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:12.785953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:12.785899    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podStartSLOduration=1.785882247 podStartE2EDuration="1.785882247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:12.785162045 +0000 UTC m=+823.769158619" watchObservedRunningTime="2026-11-03 21:29:12.785882247 +0000 UTC m=+823.769878806"
Nov 03 21:29:13.449416 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:13.449376    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:29:21.717934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:21.717898    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:21.731275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:21.731248    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:21.742648 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:21.742619    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:29:22.436432 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:29:22.436392    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:29:39.489513 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:29:39.489480    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:29:52.436088 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:29:52.436055    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:30:00.165724 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.165684    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"]
Nov 03 21:30:00.169548 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.169523    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.186759 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.186733    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"]
Nov 03 21:30:00.189985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.189970    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.224841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.224795    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.225056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.224908    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5wb6\" (UniqueName: \"kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.225056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.224978    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2m2h\" (UniqueName: \"kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.225056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.225026    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.228174 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.228150    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:30:00.271009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.270962    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"]
Nov 03 21:30:00.319385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.319286    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"]
Nov 03 21:30:00.325916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.325879    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.326105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.325979    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5wb6\" (UniqueName: \"kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.326105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.326033    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2m2h\" (UniqueName: \"kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.326105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.326065    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.326687 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.326662    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.327073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.327051    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.382417 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.382384    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5wb6\" (UniqueName: \"kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.382792 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.382769    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2m2h\" (UniqueName: \"kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h\") pod \"maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.482481 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.482442    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:00.505097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.504899    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:00.863924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.863901    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"]
Nov 03 21:30:00.866603 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:30:00.866572    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb63e8e_0484_44e5_94d0_8664273ed499.slice/crio-044d1cd3c9a2dbc1eb386b5ac0734c800636d3147b602a06e75c3e4375c49ea4 WatchSource:0}: Error finding container 044d1cd3c9a2dbc1eb386b5ac0734c800636d3147b602a06e75c3e4375c49ea4: Status 404 returned error can't find the container with id 044d1cd3c9a2dbc1eb386b5ac0734c800636d3147b602a06e75c3e4375c49ea4
Nov 03 21:30:00.908801 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.908761    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerStarted","Data":"044d1cd3c9a2dbc1eb386b5ac0734c800636d3147b602a06e75c3e4375c49ea4"}
Nov 03 21:30:00.914254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:00.914224    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"]
Nov 03 21:30:00.917388 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:30:00.917362    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf528828_526c_4b22_b13c_68dca1072b37.slice/crio-d89eff594c0a6e493afe3697ba076261a65fac4337301cc3f7e89e6bfa1a546a WatchSource:0}: Error finding container d89eff594c0a6e493afe3697ba076261a65fac4337301cc3f7e89e6bfa1a546a: Status 404 returned error can't find the container with id d89eff594c0a6e493afe3697ba076261a65fac4337301cc3f7e89e6bfa1a546a
Nov 03 21:30:01.914153 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:01.914116    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" event={"ID":"cf528828-526c-4b22-b13c-68dca1072b37","Type":"ContainerStarted","Data":"d89eff594c0a6e493afe3697ba076261a65fac4337301cc3f7e89e6bfa1a546a"}
Nov 03 21:30:06.312020 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.311952    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"]
Nov 03 21:30:06.317184 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.317161    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.368987 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.368948    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"premium-w1-f463c9-kserve-self-signed-certs\""
Nov 03 21:30:06.384131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.384102    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.384292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.384154    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.384292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.384253    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.384408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.384302    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9vg\" (UniqueName: \"kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.384450 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.384400    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.384450 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.384424    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.417705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.417660    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"]
Nov 03 21:30:06.485105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485020    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9vg\" (UniqueName: \"kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485085    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485111    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485148    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485212    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485282    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485656    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485760 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485742    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.485808 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.485763    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.487939 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.487920    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.488346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.488278    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.541725 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.541691    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9vg\" (UniqueName: \"kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg\") pod \"premium-w1-f463c9-kserve-7dd55c6554-v2tr6\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") " pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.631953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.631918    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:06.941540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:06.941508    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerStarted","Data":"5a3c70293e4f93341be58cb8a30a9f64907a02a05bc06e2bb12ada19d44939be"}
Nov 03 21:30:07.036877 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:07.036846    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"]
Nov 03 21:30:07.039805 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:30:07.039775    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0358e895_256c_43ec_9c68_38bb70e633e4.slice/crio-919fdb0c4613c2a7d866917d638e4dfbd5ee89adbf1e99f32613c46feed7e57a WatchSource:0}: Error finding container 919fdb0c4613c2a7d866917d638e4dfbd5ee89adbf1e99f32613c46feed7e57a: Status 404 returned error can't find the container with id 919fdb0c4613c2a7d866917d638e4dfbd5ee89adbf1e99f32613c46feed7e57a
Nov 03 21:30:07.947825 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:07.947787    5244 generic.go:356] "Generic (PLEG): container finished" podID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerID="5a3c70293e4f93341be58cb8a30a9f64907a02a05bc06e2bb12ada19d44939be" exitCode=60
Nov 03 21:30:07.948235 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:07.947860    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerDied","Data":"5a3c70293e4f93341be58cb8a30a9f64907a02a05bc06e2bb12ada19d44939be"}
Nov 03 21:30:07.948235 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:07.948134    5244 scope.go:122] "RemoveContainer" containerID="5a3c70293e4f93341be58cb8a30a9f64907a02a05bc06e2bb12ada19d44939be"
Nov 03 21:30:07.951745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:07.951718    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" event={"ID":"0358e895-256c-43ec-9c68-38bb70e633e4","Type":"ContainerStarted","Data":"709c5c952ac1b55e6f5d49a6df16ed008fb43fa6ed65b66a36ee6ea594fe6f5c"}
Nov 03 21:30:07.951846 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:07.951756    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" event={"ID":"0358e895-256c-43ec-9c68-38bb70e633e4","Type":"ContainerStarted","Data":"919fdb0c4613c2a7d866917d638e4dfbd5ee89adbf1e99f32613c46feed7e57a"}
Nov 03 21:30:08.176133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.176082    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" podStartSLOduration=2.176066443 podStartE2EDuration="2.176066443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:08.175419744 +0000 UTC m=+879.159416305" watchObservedRunningTime="2026-11-03 21:30:08.176066443 +0000 UTC m=+879.160063000"
Nov 03 21:30:08.632397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.632100    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:08.958544 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.958509    5244 generic.go:356] "Generic (PLEG): container finished" podID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerID="a525dd36b947382fd9553c66b49db6f237f51aaad511a609a3524d5ce5f775c4" exitCode=60
Nov 03 21:30:08.958965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.958595    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerDied","Data":"a525dd36b947382fd9553c66b49db6f237f51aaad511a609a3524d5ce5f775c4"}
Nov 03 21:30:08.958965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.958655    5244 scope.go:122] "RemoveContainer" containerID="5a3c70293e4f93341be58cb8a30a9f64907a02a05bc06e2bb12ada19d44939be"
Nov 03 21:30:08.958965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.958892    5244 scope.go:122] "RemoveContainer" containerID="a525dd36b947382fd9553c66b49db6f237f51aaad511a609a3524d5ce5f775c4"
Nov 03 21:30:08.959186 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:08.959161    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk_odh-ai-gateway-infra(7fb63e8e-0484-44e5-94d0-8664273ed499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499"
Nov 03 21:30:08.960274 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.960252    5244 generic.go:356] "Generic (PLEG): container finished" podID="cf528828-526c-4b22-b13c-68dca1072b37" containerID="8fb25f9ef587fba67d17b9f7d6e1b4b9d52e11e6def2150efadc70de79bc76f4" exitCode=60
Nov 03 21:30:08.960420 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.960293    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" event={"ID":"cf528828-526c-4b22-b13c-68dca1072b37","Type":"ContainerDied","Data":"8fb25f9ef587fba67d17b9f7d6e1b4b9d52e11e6def2150efadc70de79bc76f4"}
Nov 03 21:30:08.960581 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:08.960563    5244 scope.go:122] "RemoveContainer" containerID="8fb25f9ef587fba67d17b9f7d6e1b4b9d52e11e6def2150efadc70de79bc76f4"
Nov 03 21:30:09.974014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:09.973977    5244 generic.go:356] "Generic (PLEG): container finished" podID="cf528828-526c-4b22-b13c-68dca1072b37" containerID="f3f7b633a45a795499e9a5ce8397bc9ba43c37cc7bc84b23076f58092bc3bb21" exitCode=60
Nov 03 21:30:09.974534 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:09.974107    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" event={"ID":"cf528828-526c-4b22-b13c-68dca1072b37","Type":"ContainerDied","Data":"f3f7b633a45a795499e9a5ce8397bc9ba43c37cc7bc84b23076f58092bc3bb21"}
Nov 03 21:30:09.974534 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:09.974198    5244 scope.go:122] "RemoveContainer" containerID="8fb25f9ef587fba67d17b9f7d6e1b4b9d52e11e6def2150efadc70de79bc76f4"
Nov 03 21:30:09.974534 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:09.974474    5244 scope.go:122] "RemoveContainer" containerID="f3f7b633a45a795499e9a5ce8397bc9ba43c37cc7bc84b23076f58092bc3bb21"
Nov 03 21:30:09.974743 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:09.974719    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7_odh-ai-gateway-infra(cf528828-526c-4b22-b13c-68dca1072b37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" podUID="cf528828-526c-4b22-b13c-68dca1072b37"
Nov 03 21:30:09.975995 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:09.975977    5244 scope.go:122] "RemoveContainer" containerID="a525dd36b947382fd9553c66b49db6f237f51aaad511a609a3524d5ce5f775c4"
Nov 03 21:30:09.976175 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:09.976157    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk_odh-ai-gateway-infra(7fb63e8e-0484-44e5-94d0-8664273ed499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499"
Nov 03 21:30:10.982465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:10.982441    5244 scope.go:122] "RemoveContainer" containerID="f3f7b633a45a795499e9a5ce8397bc9ba43c37cc7bc84b23076f58092bc3bb21"
Nov 03 21:30:10.982839 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:10.982679    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7_odh-ai-gateway-infra(cf528828-526c-4b22-b13c-68dca1072b37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" podUID="cf528828-526c-4b22-b13c-68dca1072b37"
Nov 03 21:30:13.860114 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:13.860082    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:30:16.633103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:16.633064    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:16.645903 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:16.645877    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:17.023246 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:17.023218    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:30:22.129787 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:22.129753    5244 scope.go:122] "RemoveContainer" containerID="f3f7b633a45a795499e9a5ce8397bc9ba43c37cc7bc84b23076f58092bc3bb21"
Nov 03 21:30:22.130187 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:22.129864    5244 scope.go:122] "RemoveContainer" containerID="a525dd36b947382fd9553c66b49db6f237f51aaad511a609a3524d5ce5f775c4"
Nov 03 21:30:22.436223 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:22.436183    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:30:23.042106 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:23.042075    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerStarted","Data":"47fa983ce32578df75e2243c664d287bebb543bbf148bd4f56b93483078a368a"}
Nov 03 21:30:23.042545 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:23.042522    5244 scope.go:122] "RemoveContainer" containerID="47fa983ce32578df75e2243c664d287bebb543bbf148bd4f56b93483078a368a"
Nov 03 21:30:23.042814 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:23.042793    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk_odh-ai-gateway-infra(7fb63e8e-0484-44e5-94d0-8664273ed499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499"
Nov 03 21:30:23.044117 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:23.044096    5244 generic.go:356] "Generic (PLEG): container finished" podID="cf528828-526c-4b22-b13c-68dca1072b37" containerID="f48fa1562f5b05632946bcf9a295626ae7e0c2ed340c9fdc270c734d99677e68" exitCode=60
Nov 03 21:30:23.044203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:23.044173    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" event={"ID":"cf528828-526c-4b22-b13c-68dca1072b37","Type":"ContainerDied","Data":"f48fa1562f5b05632946bcf9a295626ae7e0c2ed340c9fdc270c734d99677e68"}
Nov 03 21:30:23.044247 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:23.044225    5244 scope.go:122] "RemoveContainer" containerID="f3f7b633a45a795499e9a5ce8397bc9ba43c37cc7bc84b23076f58092bc3bb21"
Nov 03 21:30:23.044584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:23.044569    5244 scope.go:122] "RemoveContainer" containerID="f48fa1562f5b05632946bcf9a295626ae7e0c2ed340c9fdc270c734d99677e68"
Nov 03 21:30:23.044810 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:23.044788    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7_odh-ai-gateway-infra(cf528828-526c-4b22-b13c-68dca1072b37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" podUID="cf528828-526c-4b22-b13c-68dca1072b37"
Nov 03 21:30:24.053296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.053259    5244 generic.go:356] "Generic (PLEG): container finished" podID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerID="47fa983ce32578df75e2243c664d287bebb543bbf148bd4f56b93483078a368a" exitCode=60
Nov 03 21:30:24.053296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.053298    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerDied","Data":"47fa983ce32578df75e2243c664d287bebb543bbf148bd4f56b93483078a368a"}
Nov 03 21:30:24.053850 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.053357    5244 scope.go:122] "RemoveContainer" containerID="a525dd36b947382fd9553c66b49db6f237f51aaad511a609a3524d5ce5f775c4"
Nov 03 21:30:24.053850 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.053696    5244 scope.go:122] "RemoveContainer" containerID="47fa983ce32578df75e2243c664d287bebb543bbf148bd4f56b93483078a368a"
Nov 03 21:30:24.053959 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:24.053940    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk_odh-ai-gateway-infra(7fb63e8e-0484-44e5-94d0-8664273ed499)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499"
Nov 03 21:30:24.162803 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.162773    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"]
Nov 03 21:30:24.276308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.274394    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"]
Nov 03 21:30:24.421308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.421284    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:24.566994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.566948    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca\") pod \"cf528828-526c-4b22-b13c-68dca1072b37\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") "
Nov 03 21:30:24.567210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.567013    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h\" (UniqueName: \"kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h\") pod \"cf528828-526c-4b22-b13c-68dca1072b37\" (UID: \"cf528828-526c-4b22-b13c-68dca1072b37\") "
Nov 03 21:30:24.567412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.567389    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca" pod "cf528828-526c-4b22-b13c-68dca1072b37" (UID: "cf528828-526c-4b22-b13c-68dca1072b37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:24.569277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.569244    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h" pod "cf528828-526c-4b22-b13c-68dca1072b37" (UID: "cf528828-526c-4b22-b13c-68dca1072b37"). InnerVolumeSpecName "kube-api-access-j2m2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:24.667920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.667840    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf528828-526c-4b22-b13c-68dca1072b37-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:24.667920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:24.667868    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2m2h\" (UniqueName: \"kubernetes.io/projected/cf528828-526c-4b22-b13c-68dca1072b37-kube-api-access-j2m2h\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:25.059638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.059608    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"
Nov 03 21:30:25.059638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.059626    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7" event={"ID":"cf528828-526c-4b22-b13c-68dca1072b37","Type":"ContainerDied","Data":"d89eff594c0a6e493afe3697ba076261a65fac4337301cc3f7e89e6bfa1a546a"}
Nov 03 21:30:25.060192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.059690    5244 scope.go:122] "RemoveContainer" containerID="f48fa1562f5b05632946bcf9a295626ae7e0c2ed340c9fdc270c734d99677e68"
Nov 03 21:30:25.197733 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.197708    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:25.223601 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.223573    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"]
Nov 03 21:30:25.225272 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.225239    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d71a74-29895690-cddz7"]
Nov 03 21:30:25.274572 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.274536    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca\") pod \"7fb63e8e-0484-44e5-94d0-8664273ed499\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") "
Nov 03 21:30:25.274749 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.274629    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6\" (UniqueName: \"kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6\") pod \"7fb63e8e-0484-44e5-94d0-8664273ed499\" (UID: \"7fb63e8e-0484-44e5-94d0-8664273ed499\") "
Nov 03 21:30:25.274983 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.274960    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca" pod "7fb63e8e-0484-44e5-94d0-8664273ed499" (UID: "7fb63e8e-0484-44e5-94d0-8664273ed499"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:30:25.276823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.276806    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6" pod "7fb63e8e-0484-44e5-94d0-8664273ed499" (UID: "7fb63e8e-0484-44e5-94d0-8664273ed499"). InnerVolumeSpecName "kube-api-access-v5wb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:30:25.376012 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.375925    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb63e8e-0484-44e5-94d0-8664273ed499-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:25.376012 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:25.375952    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5wb6\" (UniqueName: \"kubernetes.io/projected/7fb63e8e-0484-44e5-94d0-8664273ed499-kube-api-access-v5wb6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:30:26.068152 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:26.068113    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk" event={"ID":"7fb63e8e-0484-44e5-94d0-8664273ed499","Type":"ContainerDied","Data":"044d1cd3c9a2dbc1eb386b5ac0734c800636d3147b602a06e75c3e4375c49ea4"}
Nov 03 21:30:26.068152 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:26.068135    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"
Nov 03 21:30:26.068705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:26.068207    5244 scope.go:122] "RemoveContainer" containerID="47fa983ce32578df75e2243c664d287bebb543bbf148bd4f56b93483078a368a"
Nov 03 21:30:26.228471 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:26.228434    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"]
Nov 03 21:30:26.277703 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:26.277666    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895690-ptkmk"]
Nov 03 21:30:27.134788 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:27.134757    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" path="/var/lib/kubelet/pods/7fb63e8e-0484-44e5-94d0-8664273ed499/volumes"
Nov 03 21:30:27.135166 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:27.135121    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf528828-526c-4b22-b13c-68dca1072b37" path="/var/lib/kubelet/pods/cf528828-526c-4b22-b13c-68dca1072b37/volumes"
Nov 03 21:30:50.133527 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:30:50.133444    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:30:52.437739 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:30:52.437688    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:31:02.810937 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.810900    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"]
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811269    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811280    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811294    5244 cpu_manager.go:414] "containerMap: removing container" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811298    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811312    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811317    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811359    5244 cpu_manager.go:414] "containerMap: removing container" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:02.811362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811365    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:02.811624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811417    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811423    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:02.811624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811432    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811440    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:02.811624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.811445    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:02.814246 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.814229    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:02.915239 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.915204    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"e2e-conflict-a-33894f13-kserve-self-signed-certs\""
Nov 03 21:31:02.916787 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.916766    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"]
Nov 03 21:31:02.959465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.959001    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:02.959465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.959118    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:02.959465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.959181    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:02.959465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.959231    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:02.959465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.959259    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:02.959465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:02.959301    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgrg\" (UniqueName: \"kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060348 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060290    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060526 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060374    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgrg\" (UniqueName: \"kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060526 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060454    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060526 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060495    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060666 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060570    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060666 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060620    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060749    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.060860 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060827    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.061027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.060881    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.063401 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.063374    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.063519 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.063498    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.117896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.117859    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgrg\" (UniqueName: \"kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg\") pod \"e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") " pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.126055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.126021    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:03.527061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:03.527027    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"]
Nov 03 21:31:03.530433 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:31:03.530407    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb887e57d_8f61_4158_96b1_5ad5af42f9ca.slice/crio-cfd19f6d522c486fb2defd908f77a3271fe203030f1278561dd5603e4c835e4e WatchSource:0}: Error finding container cfd19f6d522c486fb2defd908f77a3271fe203030f1278561dd5603e4c835e4e: Status 404 returned error can't find the container with id cfd19f6d522c486fb2defd908f77a3271fe203030f1278561dd5603e4c835e4e
Nov 03 21:31:04.255743 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:04.255703    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" event={"ID":"b887e57d-8f61-4158-96b1-5ad5af42f9ca","Type":"ContainerStarted","Data":"114c38e91e15ed544f211f086f26d0fbf106d2c4be55618e0c840794c209c87c"}
Nov 03 21:31:04.255743 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:04.255748    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" event={"ID":"b887e57d-8f61-4158-96b1-5ad5af42f9ca","Type":"ContainerStarted","Data":"cfd19f6d522c486fb2defd908f77a3271fe203030f1278561dd5603e4c835e4e"}
Nov 03 21:31:04.370244 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:04.370186    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" podStartSLOduration=2.370166751 podStartE2EDuration="2.370166751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:04.368398359 +0000 UTC m=+935.352394946" watchObservedRunningTime="2026-11-03 21:31:04.370166751 +0000 UTC m=+935.354163310"
Nov 03 21:31:05.126177 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:05.126128    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:13.127201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:13.127102    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:13.141207 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:13.141182    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:13.317348 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:13.317308    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:31:16.496610 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.496576    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"]
Nov 03 21:31:16.497016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.496926    5244 cpu_manager.go:414] "containerMap: removing container" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:16.497016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.496937    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:16.497016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.496959    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:16.497016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.496964    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb63e8e-0484-44e5-94d0-8664273ed499" containerName="cleanup"
Nov 03 21:31:16.497145 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.497026    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf528828-526c-4b22-b13c-68dca1072b37" containerName="cleanup"
Nov 03 21:31:16.500025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.500010    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.546872 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.546840    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"]
Nov 03 21:31:16.550220 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.550205    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.589591 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589548    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.589591 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589591    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4clw\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.589786 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589649    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.589786 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589699    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.589853 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589781    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.589853 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589808    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.589919 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589877    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.589919 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589907    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.589980 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589929    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.589980 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589946    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.590045 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.589983    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.590075 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590045    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.590075 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590066    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.590136 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590082    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.590136 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590101    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.590199 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590154    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.590232 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590215    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.590262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.590241    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5bh\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.603215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.603188    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"]
Nov 03 21:31:16.661503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.661471    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"]
Nov 03 21:31:16.690763 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690724    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.690763 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690766    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4clw\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690794    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690824    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690869    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690900    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690926    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690965    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.690989    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691012    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691038    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691064    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691092    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691117    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691147    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691175    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691218    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691241    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5bh\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691281    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691281    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691897 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691613    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.691897 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691632    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.691897 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691697    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.692902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.691981    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.692902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.692156    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.692902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.692223    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.692902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.692571    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.692902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.692584    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.695055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.695030    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.695268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.695050    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.695593 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.695457    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.697278 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.697247    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.750491 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.750408    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4clw\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.750491 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.750474    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5nhq4\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.750893 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.750868    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5bh\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.764913 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.764888    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-thkfd\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:16.813463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.813425    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:16.862253 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:16.862216    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:17.243103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.243037    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"]
Nov 03 21:31:17.246100 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:31:17.246073    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fab12b_73aa_4812_b607_d27434308bc2.slice/crio-16d596b7fd10a9f930882fe9fdb96cb1eb8c015f5e8e773093fba549f54570f2 WatchSource:0}: Error finding container 16d596b7fd10a9f930882fe9fdb96cb1eb8c015f5e8e773093fba549f54570f2: Status 404 returned error can't find the container with id 16d596b7fd10a9f930882fe9fdb96cb1eb8c015f5e8e773093fba549f54570f2
Nov 03 21:31:17.248775 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.248738    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:17.248870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.248829    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:17.248937 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.248877    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:17.294789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.294762    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"]
Nov 03 21:31:17.297749 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:31:17.297722    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305d0fe0_cc53_4967_9425_175ffd72720f.slice/crio-08c04a6efe563bcfef2a36b20ce055981316991d36d8b114d9e58769a01bf08f WatchSource:0}: Error finding container 08c04a6efe563bcfef2a36b20ce055981316991d36d8b114d9e58769a01bf08f: Status 404 returned error can't find the container with id 08c04a6efe563bcfef2a36b20ce055981316991d36d8b114d9e58769a01bf08f
Nov 03 21:31:17.300207 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.300171    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:17.300306 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.300248    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:17.300306 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.300294    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:17.325997 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.325960    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" event={"ID":"305d0fe0-cc53-4967-9425-175ffd72720f","Type":"ContainerStarted","Data":"08c04a6efe563bcfef2a36b20ce055981316991d36d8b114d9e58769a01bf08f"}
Nov 03 21:31:17.329657 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.329620    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" event={"ID":"e9fab12b-73aa-4812-b607-d27434308bc2","Type":"ContainerStarted","Data":"047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7"}
Nov 03 21:31:17.329799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.329683    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" event={"ID":"e9fab12b-73aa-4812-b607-d27434308bc2","Type":"ContainerStarted","Data":"16d596b7fd10a9f930882fe9fdb96cb1eb8c015f5e8e773093fba549f54570f2"}
Nov 03 21:31:17.447715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.447650    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podStartSLOduration=1.447628524 podStartE2EDuration="1.447628524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:17.445885478 +0000 UTC m=+948.429882057" watchObservedRunningTime="2026-11-03 21:31:17.447628524 +0000 UTC m=+948.431625084"
Nov 03 21:31:17.549510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.549284    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"]
Nov 03 21:31:17.553415 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.553391    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600300 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600257    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfg79\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600309    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600446    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600493    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600619 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600575    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600704 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600688    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600719    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600750    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.600812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.600795    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.605379 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.605351    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6de8e6-openshift-default-dockercfg-vxss9\""
Nov 03 21:31:17.613475 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.613092    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"]
Nov 03 21:31:17.701897 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.701861    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.701905    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.701947    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.701971    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702005    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfg79\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702056    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702089    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702124    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702170    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702320    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.702758 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.702710    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.703261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.703142    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.703261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.703225    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.703659 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.703638    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.705035 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.705009    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.705449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.705430    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.813599 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.813515    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:17.828092 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.828057    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfg79\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.828694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.828675    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:17.868779 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:17.868743    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:18.252355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.252006    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"]
Nov 03 21:31:18.276454 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:31:18.268942    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b36ef1_e007_42ab_ac1c_ad5e6a2cd215.slice/crio-4a565123a1d2be5bed489abaed47b999d096f6a7f4be7d738f1434a99cd28cc4 WatchSource:0}: Error finding container 4a565123a1d2be5bed489abaed47b999d096f6a7f4be7d738f1434a99cd28cc4: Status 404 returned error can't find the container with id 4a565123a1d2be5bed489abaed47b999d096f6a7f4be7d738f1434a99cd28cc4
Nov 03 21:31:18.280150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.280112    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:18.280382 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.280358    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:18.280536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.280518    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:18.370652 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.370618    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" event={"ID":"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215","Type":"ContainerStarted","Data":"4a565123a1d2be5bed489abaed47b999d096f6a7f4be7d738f1434a99cd28cc4"}
Nov 03 21:31:18.385373 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.384550    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" event={"ID":"305d0fe0-cc53-4967-9425-175ffd72720f","Type":"ContainerStarted","Data":"0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8"}
Nov 03 21:31:18.504747 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.504616    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podStartSLOduration=2.504595202 podStartE2EDuration="2.504595202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:18.502310226 +0000 UTC m=+949.486306782" watchObservedRunningTime="2026-11-03 21:31:18.504595202 +0000 UTC m=+949.488591763"
Nov 03 21:31:18.815365 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.815265    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:18.815365 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.815343    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:18.873714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.872216    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:18.874109 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.874066    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:18.874307 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:18.874275    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:19.413164 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.411495    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" event={"ID":"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215","Type":"ContainerStarted","Data":"7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec"}
Nov 03 21:31:19.527624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.527568    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podStartSLOduration=2.527551795 podStartE2EDuration="2.527551795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:19.526291896 +0000 UTC m=+950.510288488" watchObservedRunningTime="2026-11-03 21:31:19.527551795 +0000 UTC m=+950.511548354"
Nov 03 21:31:19.814362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.814304    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:19.814556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.814382    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:19.867356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.863716    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:19.867356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.863791    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:19.874371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.871926    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:19.875494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.875460    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 21:31:19.875747 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.875720    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 21:31:19.899355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:19.880195    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:31:20.814940 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:20.814888    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:20.815308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:20.815239    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:20.868839 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:20.868797    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:20.869489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:20.869452    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:20.870038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:20.870014    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 21:31:20.870200 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:20.870173    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 21:31:21.815146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:21.814722    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:21.815146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:21.814780    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:21.862878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:21.862752    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:21.862878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:21.862819    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:21.872162 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:21.872132    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 21:31:21.872746 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:21.872719    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 21:31:22.535807 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:31:22.535756    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:31:22.815416 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:22.814866    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:22.815416 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:22.814946    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:22.870349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:22.870291    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 21:31:22.870638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:22.870608    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 21:31:22.871064 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:22.871045    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:22.871229 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:22.871209    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:23.391493 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.391439    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:23.822356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.820165    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:23.822356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.820227    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:23.870354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.869822    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:23.870354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.869885    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:23.870354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.870242    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 21:31:23.870354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:23.870281    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 21:31:24.814560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:24.814514    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:24.814992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:24.814592    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:24.862791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:24.862746    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:24.862961 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:24.862812    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:24.869621 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:24.869582    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 21:31:24.869768 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:24.869638    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 21:31:25.814402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.814356    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 21:31:25.814632 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.814436    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 21:31:25.862816 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.862765    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 21:31:25.862984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.862864    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 21:31:25.875102 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.875071    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:25.875560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.875534    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:25.876421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:25.876392    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 21:31:27.856517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:27.856478    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:31:27.857142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:27.857102    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:27.904489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:27.904441    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:31:27.904675 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:27.904497    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:28.899932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:28.899511    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:31:28.899932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:28.899606    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:28.955525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:28.955379    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:31:28.956073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:28.955452    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:31:29.298694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:29.298659    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:29.299247 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:29.299111    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:29.300277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:29.300255    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 21:31:29.378138 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:29.378112    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:29.378446 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:29.378426    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:29.379063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:29.379046    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:31:42.487880 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.487847    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"]
Nov 03 21:31:42.491210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.491191    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.542995 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.542962    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7ee809-openshift-default-dockercfg-qsllw\""
Nov 03 21:31:42.562288 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562258    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2b8h\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562288 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562293    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562319    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562426    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562515    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562543    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562639    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562936 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562706    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.562936 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.562721    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.595362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.595308    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"]
Nov 03 21:31:42.663735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663703    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.663956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663780    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.663956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663796    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.663956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663825    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2b8h\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.663956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663845    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.663956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663876    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.663956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663897    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663958    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.663989    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.664159    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664384 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.664199    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.664502    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664633 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.664554    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.664758 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.664738    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.666373 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.666353    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.666709 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.666677    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.720504 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.720472    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.720693 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.720670    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2b8h\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:42.804096 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:42.804009    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:43.233927 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.233884    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"]
Nov 03 21:31:43.236422 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:31:43.236386    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87e33a0_e629_4b3b_a240_7d74779cbce8.slice/crio-f487bf557ed9fd7c200c9fb0f8f7b1e636a42d90dae792e1bef8766bae70315f WatchSource:0}: Error finding container f487bf557ed9fd7c200c9fb0f8f7b1e636a42d90dae792e1bef8766bae70315f: Status 404 returned error can't find the container with id f487bf557ed9fd7c200c9fb0f8f7b1e636a42d90dae792e1bef8766bae70315f
Nov 03 21:31:43.238746 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.238707    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:43.238837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.238781    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:43.238876 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.238836    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:31:43.567015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.566925    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" event={"ID":"b87e33a0-e629-4b3b-a240-7d74779cbce8","Type":"ContainerStarted","Data":"888e0cd0c34d2bb848b45fcb0d65ff30311ee9f23ba1bd59f484ae8a231e226e"}
Nov 03 21:31:43.567015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.566992    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" event={"ID":"b87e33a0-e629-4b3b-a240-7d74779cbce8","Type":"ContainerStarted","Data":"f487bf557ed9fd7c200c9fb0f8f7b1e636a42d90dae792e1bef8766bae70315f"}
Nov 03 21:31:43.684415 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.684360    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podStartSLOduration=1.684312855 podStartE2EDuration="1.684312855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:31:43.682668579 +0000 UTC m=+974.666665162" watchObservedRunningTime="2026-11-03 21:31:43.684312855 +0000 UTC m=+974.668309414"
Nov 03 21:31:43.804746 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:43.804705    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:44.806355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:44.806305    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 21:31:44.806762 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:44.806403    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 21:31:45.805266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:45.805228    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 21:31:45.805477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:45.805288    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 21:31:46.804701 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:46.804665    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 21:31:46.805179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:46.804730    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 21:31:47.804889 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:47.804843    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 21:31:47.806736 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:47.806687    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 21:31:48.805133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:48.805093    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 21:31:48.805580 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:48.805170    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 21:31:49.805345 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:49.805284    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 03 21:31:49.805856 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:49.805377    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 03 21:31:50.809591 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:50.809558    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:50.809968 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:50.809887    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:50.810685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:50.810666    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 21:31:52.436729 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:31:52.436688    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:31:55.038314 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:31:55.038280    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:32:15.352409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:15.351457    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:32:15.352409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:15.351549    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:32:15.428214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:15.428161    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:32:15.428622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:15.428542    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:32:22.437205 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:32:22.437166    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:32:24.480561 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:24.480525    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:32:42.972594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:42.972556    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"]
Nov 03 21:32:42.976342 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:42.976308    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.112774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.112735    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.112979 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.112788    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.112979 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.112903    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.112979 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.112962    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.113156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.112984    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.113156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.113125    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxg4\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.113277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.113198    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.113277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.113243    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.113422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.113404    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.188806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.188766    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"]
Nov 03 21:32:43.214284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214247    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214290    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214551 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214501    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214551 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214535    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214694    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214797 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214692    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxg4\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214859 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214794    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214859 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214812    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.214967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.214885    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.215027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.215007    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.215084 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.215057    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.215254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.215230    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.215469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.215294    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.215469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.215399    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.217309 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.217285    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.217422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.217379    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.270880 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.270810    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.271041 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.271015    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxg4\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.288470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.288445    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:43.714738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.714706    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"]
Nov 03 21:32:43.717757 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:32:43.717726    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f12c21e_ca76_447f_9f0b_415b809f5b8c.slice/crio-34c1fa546feb591d23106ee88c57a8524c13b91bc715671cdf6350e01d776658 WatchSource:0}: Error finding container 34c1fa546feb591d23106ee88c57a8524c13b91bc715671cdf6350e01d776658: Status 404 returned error can't find the container with id 34c1fa546feb591d23106ee88c57a8524c13b91bc715671cdf6350e01d776658
Nov 03 21:32:43.720000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.719965    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:32:43.720079 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.720037    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:32:43.720079 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.720068    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:32:43.977441 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.977357    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" event={"ID":"6f12c21e-ca76-447f-9f0b-415b809f5b8c","Type":"ContainerStarted","Data":"f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd"}
Nov 03 21:32:43.977441 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:43.977434    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" event={"ID":"6f12c21e-ca76-447f-9f0b-415b809f5b8c","Type":"ContainerStarted","Data":"34c1fa546feb591d23106ee88c57a8524c13b91bc715671cdf6350e01d776658"}
Nov 03 21:32:44.093806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:44.093752    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podStartSLOduration=2.093734907 podStartE2EDuration="2.093734907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:44.092660151 +0000 UTC m=+1035.076656710" watchObservedRunningTime="2026-11-03 21:32:44.093734907 +0000 UTC m=+1035.077731535"
Nov 03 21:32:44.290406 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:44.289674    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:44.291180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:44.291149    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:44.291310 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:44.291213    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:45.289805 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:45.289772    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:45.290206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:45.289840    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:46.289086 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:46.289051    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:46.289308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:46.289105    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:47.289605 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:47.289568    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:47.290092 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:47.289632    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:48.290212 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:48.290161    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:48.290890 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:48.290840    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:49.289495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:49.289452    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:49.289712 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:49.289533    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:50.289736 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:50.289699    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 03 21:32:50.290128 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:50.289756    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 03 21:32:51.293558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:51.293531    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:51.293917 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:51.293800    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:51.294495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:51.294475    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:32:51.907120 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:32:51.907082    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:32:52.437290 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:32:52.437244    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:33:01.885663 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.885626    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"]
Nov 03 21:33:01.889378 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.889355    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.948786 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.948746    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.948955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.948794    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.948955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.948813    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.949055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.949030    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwf9\" (UniqueName: \"kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.949098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.949072    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.949159 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.949135    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:01.993098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.993064    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f463c9\"/\"distinct-2-w1-f463c9-kserve-self-signed-certs\""
Nov 03 21:33:01.994714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:01.994686    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"]
Nov 03 21:33:02.050275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050242    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050283    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050301    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050405    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwf9\" (UniqueName: \"kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050428    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050445    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050682    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050755    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.050826 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.050816    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.052720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.052700    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.053035 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.053011    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.130943 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.129895    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwf9\" (UniqueName: \"kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9\") pod \"distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") " pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.201705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.201670    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:02.592978 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:02.592932    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"]
Nov 03 21:33:02.595900 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:33:02.595871    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67a43564_488d_43e3_a765_9fda2fb7f0eb.slice/crio-6e25456117e306b47df93dcda3bc55259bf0196dec9691b87a61babfffec2bdd WatchSource:0}: Error finding container 6e25456117e306b47df93dcda3bc55259bf0196dec9691b87a61babfffec2bdd: Status 404 returned error can't find the container with id 6e25456117e306b47df93dcda3bc55259bf0196dec9691b87a61babfffec2bdd
Nov 03 21:33:03.078830 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:03.078794    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" event={"ID":"67a43564-488d-43e3-a765-9fda2fb7f0eb","Type":"ContainerStarted","Data":"bf43c22949e6eac05a57f0ea800ff1a3257243a813abcd4c1aa4f1d2d2148393"}
Nov 03 21:33:03.079247 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:03.078843    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" event={"ID":"67a43564-488d-43e3-a765-9fda2fb7f0eb","Type":"ContainerStarted","Data":"6e25456117e306b47df93dcda3bc55259bf0196dec9691b87a61babfffec2bdd"}
Nov 03 21:33:03.192209 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:03.192154    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" podStartSLOduration=2.192137798 podStartE2EDuration="2.192137798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:03.190640877 +0000 UTC m=+1054.174637459" watchObservedRunningTime="2026-11-03 21:33:03.192137798 +0000 UTC m=+1054.176134408"
Nov 03 21:33:03.202096 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:03.202060    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:12.202281 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:12.202229    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:12.215307 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:12.215276    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:13.144481 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:13.144451    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:33:15.015849 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:15.015807    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:33:22.437174 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:33:22.437132    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:33:33.588757 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.588724    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"]
Nov 03 21:33:33.594616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.594590    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.638534 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.638494    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"]
Nov 03 21:33:33.641902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.641883    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.647409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647383    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647523 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647436    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647581 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647516    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647581 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647560    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85tx\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647690 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647620    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647753 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647698    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647804 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647762    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647804 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647787    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.647894 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.647840    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.748667 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748626    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.748869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748684    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.748869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748757    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85tx\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.748869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748789    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.748869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748830    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748934    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.748978    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749030    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749049    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749059    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487mk\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749128    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749134    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749236    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749266    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749298    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749349    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749393    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749404    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749430    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749504    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749545    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.749560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749555    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.749815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.749681    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.751888 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.751864    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.752930 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.752890    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.754714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.754692    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"]
Nov 03 21:33:33.810208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.810149    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.811052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.811025    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85tx\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.813356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.813232    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"]
Nov 03 21:33:33.850598 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850519    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.850598 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850566    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.850795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850682    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487mk\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.850795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850726    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.850795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850757    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.850795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850779    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850800    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850833    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850940    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850943    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.850986    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851288 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.851223    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.851295    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.851546 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.851529    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.853301 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.853281    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.853447 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.853405    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.906734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.906705    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487mk\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.906884 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.906789    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:33.908689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.908672    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:33.955053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:33.954608    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:34.351984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.351958    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"]
Nov 03 21:33:34.354424 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:33:34.354390    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d99735e_8eea_446b_9cd9_4cb2a716ef52.slice/crio-76097fa0677e24b49f9a99fc2424a767733e5f59855591779618931336c376bd WatchSource:0}: Error finding container 76097fa0677e24b49f9a99fc2424a767733e5f59855591779618931336c376bd: Status 404 returned error can't find the container with id 76097fa0677e24b49f9a99fc2424a767733e5f59855591779618931336c376bd
Nov 03 21:33:34.356839 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.356807    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:33:34.356920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.356873    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:33:34.356920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.356900    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:33:34.404165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.404141    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"]
Nov 03 21:33:34.406877 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:33:34.406845    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfb5a45_0f96_4bd5_beb9_cec9e15c2171.slice/crio-87f84d65d4dae73f5850170dd6d6778277ad8eed88dd82d5d645eb0867d58039 WatchSource:0}: Error finding container 87f84d65d4dae73f5850170dd6d6778277ad8eed88dd82d5d645eb0867d58039: Status 404 returned error can't find the container with id 87f84d65d4dae73f5850170dd6d6778277ad8eed88dd82d5d645eb0867d58039
Nov 03 21:33:34.410275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.410241    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:33:34.410383 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.410307    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:33:34.410383 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:34.410360    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:33:35.260292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.260250    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" event={"ID":"7d99735e-8eea-446b-9cd9-4cb2a716ef52","Type":"ContainerStarted","Data":"e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557"}
Nov 03 21:33:35.260836 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.260346    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" event={"ID":"7d99735e-8eea-446b-9cd9-4cb2a716ef52","Type":"ContainerStarted","Data":"76097fa0677e24b49f9a99fc2424a767733e5f59855591779618931336c376bd"}
Nov 03 21:33:35.262514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.262485    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" event={"ID":"adfb5a45-0f96-4bd5-beb9-cec9e15c2171","Type":"ContainerStarted","Data":"152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750"}
Nov 03 21:33:35.262637 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.262535    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" event={"ID":"adfb5a45-0f96-4bd5-beb9-cec9e15c2171","Type":"ContainerStarted","Data":"87f84d65d4dae73f5850170dd6d6778277ad8eed88dd82d5d645eb0867d58039"}
Nov 03 21:33:35.443243 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.443041    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podStartSLOduration=2.44301983 podStartE2EDuration="2.44301983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:35.442955076 +0000 UTC m=+1086.426951660" watchObservedRunningTime="2026-11-03 21:33:35.44301983 +0000 UTC m=+1086.427016390"
Nov 03 21:33:35.562053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.561926    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podStartSLOduration=2.5618905830000003 podStartE2EDuration="2.561890583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:33:35.561182419 +0000 UTC m=+1086.545178976" watchObservedRunningTime="2026-11-03 21:33:35.561890583 +0000 UTC m=+1086.545887142"
Nov 03 21:33:35.909225 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.909133    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:35.910574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.910549    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 21:33:35.910671 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.910611    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 21:33:35.965913 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.957660    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:35.969352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.966496    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 21:33:35.969352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:35.966557    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 21:33:36.924410 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:36.913743    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 21:33:36.924410 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:36.913811    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 21:33:36.957767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:36.957728    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 21:33:36.958023 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:36.957996    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 21:33:37.909350 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:37.909287    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 21:33:37.909544 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:37.909374    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 21:33:37.955852 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:37.955805    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 21:33:37.956539 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:37.956507    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 21:33:38.910036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:38.910001    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 21:33:38.910223 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:38.910055    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 21:33:38.960356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:38.959756    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 21:33:38.960356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:38.959826    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 21:33:39.909965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:39.909927    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 21:33:39.910153 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:39.909986    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 21:33:39.963358 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:39.962512    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 21:33:39.963358 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:39.962584    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 21:33:40.909411 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:40.909375    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 21:33:40.909600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:40.909426    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 21:33:40.955854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:40.955810    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 21:33:40.956146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:40.956114    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 21:33:42.318588 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:42.318551    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:42.319113 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:42.318992    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:42.322920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:42.322857    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:33:42.474261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:42.474230    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:42.474574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:42.474555    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:42.475842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:42.475820    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:33:51.110938 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:51.110895    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:33:52.668834 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:33:52.668759    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:33:53.381354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:53.372425    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:53.393489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:53.393433    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:53.533356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:53.529664    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:53.578360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:53.573447    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:53.738355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:53.737460    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 21:33:54.015419 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.014446    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.141355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.133159    5244 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.313405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.312423    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.459356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.452533    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.461576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.460484    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:33:54.576527 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.576425    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.577361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.577314    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:33:54.698358 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.697527    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:33:54.779366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.775509    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:54.779366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:54.776682    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:33:55.054359 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.038982    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:55.082558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.082392    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:33:55.274355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.274247    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:55.304443 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.283304    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:33:55.304443 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.284287    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:55.315533 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.315498    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:33:55.386368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.376157    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:33:55.426354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.425916    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:33:55.440995 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.440966    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 21:33:55.453099 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.453068    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:33:55.512202 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.512175    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 21:33:55.516733 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:33:55.516707    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 21:34:00.074460 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:00.074417    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"]
Nov 03 21:34:13.682360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.681830    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"]
Nov 03 21:34:13.701353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.700101    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.851446 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.833996    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.851797 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.851765    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.851992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.851977    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.852130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.852115    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.852240 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.852226    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.886878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.886829    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.887179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.887155    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.887503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.887484    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph646\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.887701 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.887682    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:13.906355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:13.902721    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"]
Nov 03 21:34:14.020122 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.020016    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.024486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.024454    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.048565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.028774    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.075404 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.075368    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.075805 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.075761    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.076024 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.075988    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.077000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.076979    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph646\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.080451 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.080410    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.087512 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.076600    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.091933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.076919    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.092140 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.083488    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.117993 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.091471    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.118571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.118533    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.134910 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.091840    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.285451 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.285303    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph646\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.286175 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.286125    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.290694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.290608    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.317353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.293451    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.354803 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.354526    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:14.982731 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.982701    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"]
Nov 03 21:34:14.985897 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:34:14.985865    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a80c109_0e27_45d0_bb6d_a04678c1586e.slice/crio-80967f4e0f8edcb88212c186ce3474f93e5192fafd40ae79d92d646a44b454f4 WatchSource:0}: Error finding container 80967f4e0f8edcb88212c186ce3474f93e5192fafd40ae79d92d646a44b454f4: Status 404 returned error can't find the container with id 80967f4e0f8edcb88212c186ce3474f93e5192fafd40ae79d92d646a44b454f4
Nov 03 21:34:14.988223 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.988190    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:14.988291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.988260    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:14.988343 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:14.988291    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:15.095632 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.095588    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main" containerID="cri-o://114c38e91e15ed544f211f086f26d0fbf106d2c4be55618e0c840794c209c87c" gracePeriod=45
Nov 03 21:34:15.273798 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.273710    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" event={"ID":"8a80c109-0e27-45d0-bb6d-a04678c1586e","Type":"ContainerStarted","Data":"5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9"}
Nov 03 21:34:15.273798 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.273773    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" event={"ID":"8a80c109-0e27-45d0-bb6d-a04678c1586e","Type":"ContainerStarted","Data":"80967f4e0f8edcb88212c186ce3474f93e5192fafd40ae79d92d646a44b454f4"}
Nov 03 21:34:15.276907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.276871    5244 generic.go:356] "Generic (PLEG): container finished" podID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerID="114c38e91e15ed544f211f086f26d0fbf106d2c4be55618e0c840794c209c87c" exitCode=0
Nov 03 21:34:15.277056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.276903    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" event={"ID":"b887e57d-8f61-4158-96b1-5ad5af42f9ca","Type":"ContainerDied","Data":"114c38e91e15ed544f211f086f26d0fbf106d2c4be55618e0c840794c209c87c"}
Nov 03 21:34:15.355683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.355654    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:15.394069 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.394011    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podStartSLOduration=2.393996055 podStartE2EDuration="2.393996055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:15.39292085 +0000 UTC m=+1126.376917427" watchObservedRunningTime="2026-11-03 21:34:15.393996055 +0000 UTC m=+1126.377992615"
Nov 03 21:34:15.398211 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.398189    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:34:15.567086 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.566991    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs\") pod \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") "
Nov 03 21:34:15.567086 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567038    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home\") pod \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") "
Nov 03 21:34:15.567294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567217    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg\" (UniqueName: \"kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg\") pod \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") "
Nov 03 21:34:15.567294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567280    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir\") pod \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") "
Nov 03 21:34:15.567411 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567355    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home" pod "b887e57d-8f61-4158-96b1-5ad5af42f9ca" (UID: "b887e57d-8f61-4158-96b1-5ad5af42f9ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:34:15.567457 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567419    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache\") pod \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") "
Nov 03 21:34:15.567503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567458    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm\") pod \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\" (UID: \"b887e57d-8f61-4158-96b1-5ad5af42f9ca\") "
Nov 03 21:34:15.567571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567545    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir" pod "b887e57d-8f61-4158-96b1-5ad5af42f9ca" (UID: "b887e57d-8f61-4158-96b1-5ad5af42f9ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:34:15.567702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567682    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache" pod "b887e57d-8f61-4158-96b1-5ad5af42f9ca" (UID: "b887e57d-8f61-4158-96b1-5ad5af42f9ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:34:15.567910 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567848    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:15.567910 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567869    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:15.567910 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.567879    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:15.569556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.569530    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs" pod "b887e57d-8f61-4158-96b1-5ad5af42f9ca" (UID: "b887e57d-8f61-4158-96b1-5ad5af42f9ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:34:15.570067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.570047    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm" pod "b887e57d-8f61-4158-96b1-5ad5af42f9ca" (UID: "b887e57d-8f61-4158-96b1-5ad5af42f9ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:34:15.570110 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.570067    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg" pod "b887e57d-8f61-4158-96b1-5ad5af42f9ca" (UID: "b887e57d-8f61-4158-96b1-5ad5af42f9ca"). InnerVolumeSpecName "kube-api-access-qpgrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:34:15.669422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.669385    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b887e57d-8f61-4158-96b1-5ad5af42f9ca-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:15.669422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.669415    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b887e57d-8f61-4158-96b1-5ad5af42f9ca-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:15.669422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:15.669426    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgrg\" (UniqueName: \"kubernetes.io/projected/b887e57d-8f61-4158-96b1-5ad5af42f9ca-kube-api-access-qpgrg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:34:16.283093 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.283058    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"
Nov 03 21:34:16.283093 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.283086    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx" event={"ID":"b887e57d-8f61-4158-96b1-5ad5af42f9ca","Type":"ContainerDied","Data":"cfd19f6d522c486fb2defd908f77a3271fe203030f1278561dd5603e4c835e4e"}
Nov 03 21:34:16.283617 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.283136    5244 scope.go:122] "RemoveContainer" containerID="114c38e91e15ed544f211f086f26d0fbf106d2c4be55618e0c840794c209c87c"
Nov 03 21:34:16.358674 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.358636    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:34:16.358859 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.358705    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 21:34:16.468497 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.468464    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"]
Nov 03 21:34:16.517610 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:16.517578    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/e2e-conflict-a-33894f13-kserve-6448d6c7cd-rx9nx"]
Nov 03 21:34:17.158776 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:17.158737    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" path="/var/lib/kubelet/pods/b887e57d-8f61-4158-96b1-5ad5af42f9ca/volumes"
Nov 03 21:34:17.356483 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:17.356439    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:34:17.356888 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:17.356506    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 21:34:18.356815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:18.356779    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:34:18.357313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:18.356853    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 21:34:19.356353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.355747    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:34:19.356353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.355817    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 21:34:19.454356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.452627    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"]
Nov 03 21:34:19.454356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.453265    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main"
Nov 03 21:34:19.454356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.453293    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main"
Nov 03 21:34:19.456558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.456534    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b887e57d-8f61-4158-96b1-5ad5af42f9ca" containerName="main"
Nov 03 21:34:19.459899 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.459879    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.513871 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.513839    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"]
Nov 03 21:34:19.561719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.561620    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d71a74\"/\"distinct-w2-d71a74-kserve-self-signed-certs\""
Nov 03 21:34:19.673063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.672977    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvdpv\" (UniqueName: \"kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.673421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.673400    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.673611 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.673597    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.673856 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.673842    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.674015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.674000    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.674139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.674126    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.775224 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.775184    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.775444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.775238    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.775444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.775256    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.775444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.775291    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvdpv\" (UniqueName: \"kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.775444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.775339    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.775444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.775361    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.776039 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.776015    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.776230 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.776094    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.776805 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.776785    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.778730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.778708    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.779291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.779271    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:19.864221 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:19.864185    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvdpv\" (UniqueName: \"kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv\") pod \"distinct-w2-d71a74-kserve-97765c459-9rg4q\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") " pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:20.073191 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:20.073149    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:20.358150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:20.357123    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:34:20.358150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:20.357186    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 21:34:20.473097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:20.472968    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"]
Nov 03 21:34:20.475764 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:34:20.475737    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod715ce125_f53c_4b2e_848e_029866cbc5cf.slice/crio-e7ac18da9fb3a39b5932e634d116b3eb400e300a0faca1843054c665868df542 WatchSource:0}: Error finding container e7ac18da9fb3a39b5932e634d116b3eb400e300a0faca1843054c665868df542: Status 404 returned error can't find the container with id e7ac18da9fb3a39b5932e634d116b3eb400e300a0faca1843054c665868df542
Nov 03 21:34:21.311186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:21.311153    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" event={"ID":"715ce125-f53c-4b2e-848e-029866cbc5cf","Type":"ContainerStarted","Data":"0de62eb2a43f31ac9cdaec25c7027cbaa10b801ccfbc6721510b2b9793ec379f"}
Nov 03 21:34:21.311186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:21.311192    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" event={"ID":"715ce125-f53c-4b2e-848e-029866cbc5cf","Type":"ContainerStarted","Data":"e7ac18da9fb3a39b5932e634d116b3eb400e300a0faca1843054c665868df542"}
Nov 03 21:34:21.355541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:21.355498    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 21:34:21.355736 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:21.355572    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 21:34:21.424015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:21.423952    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" podStartSLOduration=2.423936209 podStartE2EDuration="2.423936209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:21.423603108 +0000 UTC m=+1132.407599713" watchObservedRunningTime="2026-11-03 21:34:21.423936209 +0000 UTC m=+1132.407932770"
Nov 03 21:34:22.073715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:22.073682    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:22.361300 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:22.361216    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:22.361647 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:22.361625    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:22.362549 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:22.362519    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:34:22.438629 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:34:22.438585    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:34:30.073890 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:30.073842    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:30.086929 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:30.086900    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:30.372405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:30.372278    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 21:34:31.520430 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:31.520399    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:34:40.992972 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:40.992932    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"]
Nov 03 21:34:40.998167 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:40.998141    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.088592 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.088551    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.088592 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.088595    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l29k\" (UniqueName: \"kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.088813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.088691    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.088813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.088734    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.088908 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.088895    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.088943 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.088925    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.100736 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.100695    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"distinct-w0-7ee809-kserve-self-signed-certs\""
Nov 03 21:34:41.102623 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.102591    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"]
Nov 03 21:34:41.190093 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.190034    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.190510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.190113    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.190510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.190200    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l29k\" (UniqueName: \"kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.190510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.190263    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.190510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.190297    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.190510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.190481    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.191280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.191254    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.191463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.191441    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.191463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.191449    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.193412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.193391    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.193724 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.193706    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.298013 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.297940    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l29k\" (UniqueName: \"kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k\") pod \"distinct-w0-7ee809-kserve-647947879f-ft7j8\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.310550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.310489    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:41.824724 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:41.824540    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"]
Nov 03 21:34:42.442686 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:42.442642    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" event={"ID":"44ef4c6b-0630-4abb-b04b-0d2e455d8b28","Type":"ContainerStarted","Data":"f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f"}
Nov 03 21:34:42.443180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:42.442698    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" event={"ID":"44ef4c6b-0630-4abb-b04b-0d2e455d8b28","Type":"ContainerStarted","Data":"4c2745f34dcc4429aff36560a903b7bd7776f246ce1f121d4fbe74592313c023"}
Nov 03 21:34:42.555076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:42.555026    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" podStartSLOduration=2.555007913 podStartE2EDuration="2.555007913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:42.554542142 +0000 UTC m=+1153.538538701" watchObservedRunningTime="2026-11-03 21:34:42.555007913 +0000 UTC m=+1153.539004472"
Nov 03 21:34:43.311036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:43.311000    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:48.716357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:48.716193    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:34:49.370670 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.370627    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"]
Nov 03 21:34:49.374302 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.374275    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378044 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378014    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnhh\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378187 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378063    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378263 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378217    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378263 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378252    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378363    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378405    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378518 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378470    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378550    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.378638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.378608    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.426235 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.426200    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f463c9-openshift-default-dockercfg-jbvbq\""
Nov 03 21:34:49.479417 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479380    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479449    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479478    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479539    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479582    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479620    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479715    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479742    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"]
Nov 03 21:34:49.479906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479744    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.479906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.479878    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnhh\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.480122 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.480004    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.480260 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.480239    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.480519 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.480494    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.480618 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.480556    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.481120 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.481094    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.482945 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.482922    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.483711 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.483689    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.539230 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.539187    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.539410 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.539313    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnhh\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:49.691599 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:49.691563    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:50.129594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.129569    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"]
Nov 03 21:34:50.131889 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:34:50.131854    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad382be0_f946_4823_855a_1bed3a5e9c84.slice/crio-49f3928c503662c4e9839846ead3ad48e8f09112c85120d029afd67eb99a40e7 WatchSource:0}: Error finding container 49f3928c503662c4e9839846ead3ad48e8f09112c85120d029afd67eb99a40e7: Status 404 returned error can't find the container with id 49f3928c503662c4e9839846ead3ad48e8f09112c85120d029afd67eb99a40e7
Nov 03 21:34:50.134487 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.134448    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:50.134565 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.134533    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:50.134603 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.134573    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:50.554821 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.554782    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" event={"ID":"ad382be0-f946-4823-855a-1bed3a5e9c84","Type":"ContainerStarted","Data":"02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75"}
Nov 03 21:34:50.555138 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.554853    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" event={"ID":"ad382be0-f946-4823-855a-1bed3a5e9c84","Type":"ContainerStarted","Data":"49f3928c503662c4e9839846ead3ad48e8f09112c85120d029afd67eb99a40e7"}
Nov 03 21:34:50.672194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.672139    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podStartSLOduration=1.67212293 podStartE2EDuration="1.67212293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:50.669732175 +0000 UTC m=+1161.653728734" watchObservedRunningTime="2026-11-03 21:34:50.67212293 +0000 UTC m=+1161.656119567"
Nov 03 21:34:50.691804 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:50.691777    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:51.311026 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:51.310991    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:51.324170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:51.324147    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:51.572960 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:51.572876    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:34:51.693239 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:51.693208    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:34:51.693456 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:51.693261    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 21:34:52.468522 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:34:52.468487    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:34:52.692052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:52.692005    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:34:52.692247 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:52.692080    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 21:34:53.692388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:53.692316    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:34:53.692754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:53.692415    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 21:34:54.692137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:54.692095    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:34:54.692366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:54.692174    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 21:34:55.692490 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:55.692452    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:34:55.692915 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:55.692524    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 21:34:56.692060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:56.692024    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 21:34:56.692242 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:56.692090    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 21:34:57.108768 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.108676    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"]
Nov 03 21:34:57.112778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.112755    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.164689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.164656    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d71a74-openshift-default-dockercfg-nb5n6\""
Nov 03 21:34:57.203961 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.203923    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.203961 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.203965    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204066    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204190    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trsk\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204272    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204374    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204547 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204425    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204547 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204482    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.204547 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.204541    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.216014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.215987    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"]
Nov 03 21:34:57.305791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.305757    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.305791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.305798    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.305839    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.305897    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trsk\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.305971    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306008    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306047    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306083    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306142    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306481 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306393    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306481 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306459    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306547    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.306979 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.306954    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.307255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.307084    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.308478 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.308448    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.309086 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.309064    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.363229 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.363154    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.363519 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.363494    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trsk\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.426224 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.426187    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:57.736284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.736257    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:57.737020 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.736991    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:57.738163 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.738135    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:34:57.856240 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.856204    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"]
Nov 03 21:34:57.899899 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:34:57.899871    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5067398_2831_4257_b8a0_35c66bd9b3d4.slice/crio-0df7aab64401317ce67b6db2b50dfe05e9e37946db283ce09a4e81ac824f084f WatchSource:0}: Error finding container 0df7aab64401317ce67b6db2b50dfe05e9e37946db283ce09a4e81ac824f084f: Status 404 returned error can't find the container with id 0df7aab64401317ce67b6db2b50dfe05e9e37946db283ce09a4e81ac824f084f
Nov 03 21:34:57.902799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.902757    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:57.902928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.902832    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:57.902928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:57.902863    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:34:58.604015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:58.603980    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" event={"ID":"d5067398-2831-4257-b8a0-35c66bd9b3d4","Type":"ContainerStarted","Data":"fad81bc23f969c6c46974fe749748ab6a12cff6945dbf77a507a48c8170c11f9"}
Nov 03 21:34:58.604445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:58.604036    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" event={"ID":"d5067398-2831-4257-b8a0-35c66bd9b3d4","Type":"ContainerStarted","Data":"0df7aab64401317ce67b6db2b50dfe05e9e37946db283ce09a4e81ac824f084f"}
Nov 03 21:34:58.721431 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:58.721381    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podStartSLOduration=1.721366572 podStartE2EDuration="1.721366572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:58.719507757 +0000 UTC m=+1169.703504346" watchObservedRunningTime="2026-11-03 21:34:58.721366572 +0000 UTC m=+1169.705363131"
Nov 03 21:34:59.426435 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:59.426368    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:34:59.428092 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:59.428058    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:34:59.428226 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:34:59.428115    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:35:00.426838 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:00.426801    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:35:00.427319 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:00.426863    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:35:01.426610 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:01.426575    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:35:01.426813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:01.426641    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:35:02.426585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:02.426549    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:35:02.427089 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:02.426611    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:35:03.426822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:03.426782    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:35:03.427239 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:03.426849    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:35:04.426601 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:04.426560    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 21:35:04.426813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:04.426645    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 21:35:05.431211 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:05.431176    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:35:05.431673 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:05.431471    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:35:05.432089 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:05.432071    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:35:10.673542 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:10.673465    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:35:19.475195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.475159    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"]
Nov 03 21:35:19.479219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.479196    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.531102 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.531063    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-da5f0b-openshift-default-dockercfg-nvv9f\""
Nov 03 21:35:19.582714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.582679    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"]
Nov 03 21:35:19.642794 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.642754    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643007 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.642821    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643007 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.642927    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643007 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.642956    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643110 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.643038    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643110 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.643088    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643173 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.643108    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjnd\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643209 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.643188    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.643244 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.643216    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.744908    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.744962    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.744988    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745011    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745041    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745069    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745094    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnd\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745147    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745185    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745444    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745466    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745552    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745720    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.745994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.745968    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.747513 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.747492    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.747781 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.747764    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.801439 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.801408    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:19.801594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:19.801484    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjnd\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd\") pod \"e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") " pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:20.093870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.093772    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:20.519406 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.519366    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"]
Nov 03 21:35:20.561946 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:35:20.561906    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01945076_86c3_42f5_bf87_8cfbe2950a36.slice/crio-72c163ad30e1cf6fc473227323ba2eb22a335ee9f348092e7a00935048bab4cf WatchSource:0}: Error finding container 72c163ad30e1cf6fc473227323ba2eb22a335ee9f348092e7a00935048bab4cf: Status 404 returned error can't find the container with id 72c163ad30e1cf6fc473227323ba2eb22a335ee9f348092e7a00935048bab4cf
Nov 03 21:35:20.564675 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.564635    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:35:20.564760 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.564737    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:35:20.564798 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.564777    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:35:20.816791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.816702    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" event={"ID":"01945076-86c3-42f5-bf87-8cfbe2950a36","Type":"ContainerStarted","Data":"6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397"}
Nov 03 21:35:20.816930 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.816801    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" event={"ID":"01945076-86c3-42f5-bf87-8cfbe2950a36","Type":"ContainerStarted","Data":"72c163ad30e1cf6fc473227323ba2eb22a335ee9f348092e7a00935048bab4cf"}
Nov 03 21:35:20.934036 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:20.933964    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podStartSLOduration=1.933944127 podStartE2EDuration="1.933944127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:20.930778974 +0000 UTC m=+1191.914775627" watchObservedRunningTime="2026-11-03 21:35:20.933944127 +0000 UTC m=+1191.917940686"
Nov 03 21:35:21.094518 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:21.094437    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:21.095709 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:21.095686    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:21.095809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:21.095745    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:22.094254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:22.094219    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:22.094744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:22.094273    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:22.466992 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:35:22.466950    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:35:23.095011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:23.094975    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:23.095567 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:23.095032    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:24.094434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:24.094401    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:24.094639 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:24.094457    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:25.094178 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:25.094144    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:25.094601 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:25.094201    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:26.094313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:26.094272    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:26.094781 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:26.094352    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:27.107366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:27.104780    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:35:27.107366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:27.104890    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:35:28.112600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:28.112563    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:28.113395 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:28.113374    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:28.114737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:28.114718    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:35:32.696355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:32.695754    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:32.726357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:32.722411    5244 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 21:35:33.176898 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.176787    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.204528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.204480    5244 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.472356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.453032    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.618100 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.522471    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.618799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.548857    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.619034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.617094    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:33.731412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.729167    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 21:35:33.774898 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:33.741605    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:35:34.179073 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.178956    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:34.196543 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.196500    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:35:34.453806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.451687    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:34.459103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.459064    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" podUID="715ce125-f53c-4b2e-848e-029866cbc5cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:34.535283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.535238    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:34.618967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.618916    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:35:34.619647 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.619626    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:35:34.672093 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.672027    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"]
Nov 03 21:35:34.708846 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.708762    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.751353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.732251    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"]
Nov 03 21:35:34.831087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.753104    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Nov 03 21:35:34.831087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.790685    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 21:35:34.831087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.790717    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Nov 03 21:35:34.831087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.818967    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:35:34.831087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.819024    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:35:34.837285 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.791409    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.837634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.837607    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.837898 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.837871    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.838052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.838034    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.838175 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.838156    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.838313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.838296    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282kx\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.838513 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.838493    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.838660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.838645    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.838823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.838808    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.898523 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.898479    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 21:35:34.917600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.900995    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 21:35:34.917600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.904406    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 21:35:34.998197 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.998106    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.998710 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.998494    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.998980 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.998952    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.999114 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.999100    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.999216 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.999204    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.999320 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.999307    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282kx\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.999468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.999455    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.999587 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.999575    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:34.999738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.999724    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.000256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.000230    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 21:35:35.014349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.000597    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.014509 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:34.998834    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.019008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.017164    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.019008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.018186    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.019008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.018587    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.028714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.021477    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 21:35:35.028714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.027958    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.046896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.046864    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.080716 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.080647    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-282kx\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.098869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.098725    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token\") pod \"e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") " pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.153880 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.153584    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:35.659653 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.659623    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"]
Nov 03 21:35:35.661487 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:35:35.661463    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb02b475_191c_4dc5_b815_33afa4f756b6.slice/crio-afeab1d48e7fa58dc3b4dca289facaaa08b004ef0ee9e6e088a60e7649029e07 WatchSource:0}: Error finding container afeab1d48e7fa58dc3b4dca289facaaa08b004ef0ee9e6e088a60e7649029e07: Status 404 returned error can't find the container with id afeab1d48e7fa58dc3b4dca289facaaa08b004ef0ee9e6e088a60e7649029e07
Nov 03 21:35:35.664052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.664015    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:35:35.664131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.664095    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:35:35.664168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:35.664132    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:35:36.271176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:36.271132    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" event={"ID":"bb02b475-191c-4dc5-b815-33afa4f756b6","Type":"ContainerStarted","Data":"dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd"}
Nov 03 21:35:36.271605 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:36.271225    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" event={"ID":"bb02b475-191c-4dc5-b815-33afa4f756b6","Type":"ContainerStarted","Data":"afeab1d48e7fa58dc3b4dca289facaaa08b004ef0ee9e6e088a60e7649029e07"}
Nov 03 21:35:36.386786 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:36.386723    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podStartSLOduration=2.386709051 podStartE2EDuration="2.386709051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:36.385721026 +0000 UTC m=+1207.369717585" watchObservedRunningTime="2026-11-03 21:35:36.386709051 +0000 UTC m=+1207.370705609"
Nov 03 21:35:37.158012 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:37.157976    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 21:35:37.158201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:37.158036    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 21:35:37.161309 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:37.161283    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:38.156795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:38.156757    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 21:35:38.157263 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:38.156818    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 21:35:39.156912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:39.156866    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 21:35:39.157521 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:39.156934    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 21:35:40.156832 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:40.156769    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 21:35:40.157034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:40.156855    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 21:35:41.157344 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:41.157285    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 21:35:41.157845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:41.157379    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 21:35:42.154512 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:42.154473    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 21:35:42.154695 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:42.154531    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 21:35:43.159089 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:43.159058    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:43.159566 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:43.159368    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:43.160116 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:43.160101    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:35:44.578554 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:35:44.578515    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:35:52.437496 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:35:52.437453    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:36:08.554483 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.554434    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"]
Nov 03 21:36:08.559746 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.559720    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.616634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.616601    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.616841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.616655    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.616841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.616753    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.616841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.616820    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.616959 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.616927    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.617027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.617010    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.641090 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.641049    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7ee809\"/\"distinct-2-w0-7ee809-kserve-self-signed-certs\""
Nov 03 21:36:08.644744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.644712    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"]
Nov 03 21:36:08.718577 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718541    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.718783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718593    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.718783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718641    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.718783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718672    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.718783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718716    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.719021 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718858    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.719021 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.718986    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.719181 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.719161    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.719228 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.719174    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.721342 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.721299    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.721642 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.721620    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.805400 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.805286    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt\") pod \"distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") " pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:08.876668 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:08.876633    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:09.559998 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:09.559891    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"]
Nov 03 21:36:09.569451 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:36:09.569413    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83390212_04c2_49ff_824d_8cb84bb7c9ac.slice/crio-544deba14b30bf8f15c2ff5f143313c50ed92e856191395647d48072a79f4157 WatchSource:0}: Error finding container 544deba14b30bf8f15c2ff5f143313c50ed92e856191395647d48072a79f4157: Status 404 returned error can't find the container with id 544deba14b30bf8f15c2ff5f143313c50ed92e856191395647d48072a79f4157
Nov 03 21:36:10.564355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:10.563394    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7" event={"ID":"83390212-04c2-49ff-824d-8cb84bb7c9ac","Type":"ContainerStarted","Data":"934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35"}
Nov 03 21:36:10.564355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:10.563447    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7" event={"ID":"83390212-04c2-49ff-824d-8cb84bb7c9ac","Type":"ContainerStarted","Data":"544deba14b30bf8f15c2ff5f143313c50ed92e856191395647d48072a79f4157"}
Nov 03 21:36:10.879411 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:10.878565    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:18.879174 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:18.879137    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:18.891733 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:18.891702    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:19.010781 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:19.010731    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7" podStartSLOduration=11.010718209 podStartE2EDuration="11.010718209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:10.691630388 +0000 UTC m=+1241.675626948" watchObservedRunningTime="2026-11-03 21:36:19.010718209 +0000 UTC m=+1249.994714767"
Nov 03 21:36:19.651832 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:19.651802    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 21:36:22.437652 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:36:22.437588    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:36:31.409009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:36:31.408972    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:36:52.480505 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:36:52.480457    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:37:11.953608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:11.922117    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:37:13.148869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.148842    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"]
Nov 03 21:37:13.152802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.152781    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.211809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.211778    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"]
Nov 03 21:37:13.261479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261440    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261500    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261532    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261586    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5g8\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261675    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261706    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261732    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.261924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261756    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.262089 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.261996    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363058 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363020    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363071    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363100    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363128    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363181    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363452    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363537    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363549    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.363911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363776    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.364052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363929    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.364087 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.363492    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.364139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.364118    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.364188 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.364154    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.364355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.364163    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5g8\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.365912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.365884    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.366732 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.366706    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.420760 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.420685    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.421033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.421007    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5g8\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.468270 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.468231    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:13.899380 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.899353    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"]
Nov 03 21:37:13.901417 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:37:13.901385    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30365623_40de_4ff3_937c_ecea7cb4f62d.slice/crio-c1e3702eb32c37c41c868263badd385dd0eeb2a4540f06c2d18dc589783ab654 WatchSource:0}: Error finding container c1e3702eb32c37c41c868263badd385dd0eeb2a4540f06c2d18dc589783ab654: Status 404 returned error can't find the container with id c1e3702eb32c37c41c868263badd385dd0eeb2a4540f06c2d18dc589783ab654
Nov 03 21:37:13.904443 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.904406    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:13.904528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.904492    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:13.904528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:13.904521    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:14.085498 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:14.085463    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" event={"ID":"30365623-40de-4ff3-937c-ecea7cb4f62d","Type":"ContainerStarted","Data":"37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb"}
Nov 03 21:37:14.085688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:14.085530    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" event={"ID":"30365623-40de-4ff3-937c-ecea7cb4f62d","Type":"ContainerStarted","Data":"c1e3702eb32c37c41c868263badd385dd0eeb2a4540f06c2d18dc589783ab654"}
Nov 03 21:37:14.202262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:14.202202    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podStartSLOduration=1.202187074 podStartE2EDuration="1.202187074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:14.199120942 +0000 UTC m=+1305.183117525" watchObservedRunningTime="2026-11-03 21:37:14.202187074 +0000 UTC m=+1305.186183634"
Nov 03 21:37:14.468740 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:14.468641    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:14.470233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:14.470200    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:14.470374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:14.470262    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:15.468736 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:15.468698    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:15.469139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:15.468751    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:16.468963 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:16.468923    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:16.469436 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:16.468979    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:17.469142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:17.469103    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:17.469616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:17.469157    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:18.469115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:18.469082    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:18.469298 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:18.469136    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:19.468790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:19.468756    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:19.468979 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:19.468820    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:20.469151 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:20.469116    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 21:37:20.469582 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:20.469188    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 21:37:21.473293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:21.473256    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:21.473804 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:21.473690    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:21.474514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:21.474498    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:37:22.438657 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:37:22.438618    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:37:28.285165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.285123    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"]
Nov 03 21:37:28.289319 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.289296    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.398785 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.398750    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"]
Nov 03 21:37:28.448504 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448456    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448527    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448622    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448673    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hbl\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448740    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448817    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448869    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.448958 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448911    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.449005 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.448988    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550569    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550645    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550905 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550679    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550905 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550783    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550905 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550831    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550905 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550857    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.550905 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550897    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hbl\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550935    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.550992    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.551084    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.551139    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.551470    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.551499    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.551587 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.551559    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.553374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.553347    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.553526 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.553508    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.607823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.607785    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.607967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.607841    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hbl\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl\") pod \"e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") " pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:28.902830 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:28.902743    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:29.332112 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:29.332088    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"]
Nov 03 21:37:29.333751 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:37:29.333720    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f75301_9554_4ea7_b78f_2bab94f03ef7.slice/crio-81947f0e1cb9d311ac3b713cf0977be8f4de7d5a1abb77c9e00b4b20497531f5 WatchSource:0}: Error finding container 81947f0e1cb9d311ac3b713cf0977be8f4de7d5a1abb77c9e00b4b20497531f5: Status 404 returned error can't find the container with id 81947f0e1cb9d311ac3b713cf0977be8f4de7d5a1abb77c9e00b4b20497531f5
Nov 03 21:37:29.336182 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:29.336147    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:29.336274 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:29.336212    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:29.336274 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:29.336241    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:30.228304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:30.228267    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" event={"ID":"f0f75301-9554-4ea7-b78f-2bab94f03ef7","Type":"ContainerStarted","Data":"0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7"}
Nov 03 21:37:30.228505 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:30.228320    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" event={"ID":"f0f75301-9554-4ea7-b78f-2bab94f03ef7","Type":"ContainerStarted","Data":"81947f0e1cb9d311ac3b713cf0977be8f4de7d5a1abb77c9e00b4b20497531f5"}
Nov 03 21:37:30.393796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:30.393736    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podStartSLOduration=2.393719429 podStartE2EDuration="2.393719429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:30.392219292 +0000 UTC m=+1321.376215870" watchObservedRunningTime="2026-11-03 21:37:30.393719429 +0000 UTC m=+1321.377715990"
Nov 03 21:37:30.903493 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:30.903447    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:30.905471 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:30.905443    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 21:37:30.905650 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:30.905495    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 21:37:31.904044 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:31.904002    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 21:37:31.904536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:31.904071    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 21:37:32.903486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:32.903443    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 21:37:32.903710 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:32.903501    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 21:37:33.731268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.731232    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"]
Nov 03 21:37:33.758802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.758772    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808296    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808419    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808465    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808494    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808562    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808592    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808691    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808720    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.808761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.808742    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lrqs\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.810669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.810651    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c1662df5-openshift-default-dockercfg-g9tfw\""
Nov 03 21:37:33.864362 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.864304    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"]
Nov 03 21:37:33.904657 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.904622    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 21:37:33.904951 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.904917    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 21:37:33.910533 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910501    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910556    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910590    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910619    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrqs\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910833 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910748    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910833 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910803    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910935 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910853    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.910935 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910899    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.911042 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.910980    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.911388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.911366    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.911602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.911584    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.912053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.912032    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.912180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.912074    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.912797 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.912778    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.915192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.915172    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.916284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.916259    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.968587 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.968559    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:33.968813 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:33.968784    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrqs\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:34.072910 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.072818    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:34.553355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.549503    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"]
Nov 03 21:37:34.574585 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:37:34.574554    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5557439_52d9_4821_b6e3_e8df9649a051.slice/crio-ccf50d5dbc3a392ca7b877fefa2cc4094b8c88b15970ee3324cf998bbd8677b5 WatchSource:0}: Error finding container ccf50d5dbc3a392ca7b877fefa2cc4094b8c88b15970ee3324cf998bbd8677b5: Status 404 returned error can't find the container with id ccf50d5dbc3a392ca7b877fefa2cc4094b8c88b15970ee3324cf998bbd8677b5
Nov 03 21:37:34.577374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.577154    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:34.577374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.577223    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:34.577374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.577266    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:37:34.904318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.904223    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 21:37:34.904318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:34.904284    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 21:37:35.282084 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:35.282046    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" event={"ID":"c5557439-52d9-4821-b6e3-e8df9649a051","Type":"ContainerStarted","Data":"b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673"}
Nov 03 21:37:35.282304 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:35.282110    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" event={"ID":"c5557439-52d9-4821-b6e3-e8df9649a051","Type":"ContainerStarted","Data":"ccf50d5dbc3a392ca7b877fefa2cc4094b8c88b15970ee3324cf998bbd8677b5"}
Nov 03 21:37:35.416512 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:35.416456    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podStartSLOduration=2.416440201 podStartE2EDuration="2.416440201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:37:35.407815239 +0000 UTC m=+1326.391811863" watchObservedRunningTime="2026-11-03 21:37:35.416440201 +0000 UTC m=+1326.400436801"
Nov 03 21:37:35.904061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:35.904014    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 21:37:35.904296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:35.904083    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 21:37:36.073730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:36.073686    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:36.075099 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:36.075072    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 21:37:36.075242 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:36.075116    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 21:37:36.917663 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:36.917612    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:36.918068 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:36.918033    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:36.919222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:36.919201    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:37:37.073289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:37.073244    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 21:37:37.073494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:37.073322    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 21:37:38.073435 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:38.073386    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 21:37:38.073922 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:38.073452    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 21:37:39.073816 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:39.073780    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 21:37:39.074284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:39.073843    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 21:37:40.073810 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:40.073775    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 21:37:40.074457 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:40.074407    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 21:37:40.910517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:40.910461    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:37:41.008722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:41.007448    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:37:41.080976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:41.080501    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 21:37:41.080976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:41.080570    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 21:37:42.189701 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:42.189665    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:42.190670 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:42.190645    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:42.191955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:37:42.191935    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:37:52.437167 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:37:52.437118    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:38:01.178091 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:01.178053    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:38:02.473809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.473773    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"]
Nov 03 21:38:02.478041 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.478011    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717153    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717235    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717290    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717316    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717367    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717401    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717468    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7dd\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717590    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.717642    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.724408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.718760    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-30f143-openshift-default-dockercfg-x44m8\""
Nov 03 21:38:02.781444 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.781403    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"]
Nov 03 21:38:02.831206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.831160    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.844513 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.844477    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.863029 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.858813    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.863420 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.831810    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.876636 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.876603    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.876913 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.876893    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.877477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.877365    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.877729 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.877708    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.882064 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.881443    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7dd\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.897142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.897112    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.913637 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.878081    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.913889 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.891771    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.914023 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.909708    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.914161 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.914140    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.914425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.914402    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.914615 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.914433    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.949683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.949652    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7dd\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:02.970519 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:02.970477    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token\") pod \"e2e-api-a-30f143-openshift-default-79884b6775-fjbnr\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") " pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:03.099113 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.099002    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:03.613024 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.612998    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"]
Nov 03 21:38:03.615596 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:03.615566    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5e5831_49d4_4fc9_adf7_73177d082a13.slice/crio-bc43f320cf2f430dfb66496762cb828a9a7c202e228ad21642f0ef4adf4a47b1 WatchSource:0}: Error finding container bc43f320cf2f430dfb66496762cb828a9a7c202e228ad21642f0ef4adf4a47b1: Status 404 returned error can't find the container with id bc43f320cf2f430dfb66496762cb828a9a7c202e228ad21642f0ef4adf4a47b1
Nov 03 21:38:03.618210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.618171    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:38:03.618316 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.618243    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:38:03.618316 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.618273    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:38:03.786395 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.786352    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" event={"ID":"ae5e5831-49d4-4fc9-adf7-73177d082a13","Type":"ContainerStarted","Data":"2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"}
Nov 03 21:38:03.786592 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.786419    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" event={"ID":"ae5e5831-49d4-4fc9-adf7-73177d082a13","Type":"ContainerStarted","Data":"bc43f320cf2f430dfb66496762cb828a9a7c202e228ad21642f0ef4adf4a47b1"}
Nov 03 21:38:03.903299 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:03.903178    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podStartSLOduration=1.903155626 podStartE2EDuration="1.903155626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:03.900414626 +0000 UTC m=+1354.884411211" watchObservedRunningTime="2026-11-03 21:38:03.903155626 +0000 UTC m=+1354.887152187"
Nov 03 21:38:04.106355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:04.104629    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:04.106355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:04.104779    5244 patch_prober.go:29] interesting pod/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 21:38:04.106355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:04.104844    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 21:38:05.104824 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:05.104793    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:05.797215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:05.797183    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:05.798290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:05.798270    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:38:10.450409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.449967    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"]
Nov 03 21:38:10.466352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.455279    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.492313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.490693    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgxl\" (UniqueName: \"kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl\") pod \"payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") " pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.492313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.490966    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") " pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.543363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.510918    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"]
Nov 03 21:38:10.555412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.545792    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-30f143-dockercfg-6nqqd\""
Nov 03 21:38:10.570131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.546063    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-30f143\""
Nov 03 21:38:10.624568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.624537    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") " pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.629353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.629302    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgxl\" (UniqueName: \"kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl\") pod \"payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") " pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.629584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.625624    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") " pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.685776 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.685749    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgxl\" (UniqueName: \"kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl\") pod \"payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") " pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:10.878555 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:10.878474    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:11.256718 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:11.256692    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"]
Nov 03 21:38:11.259657 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:11.259630    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea56e23a_3305_47a8_b1a1_51c222c2680e.slice/crio-43bcb6471539c49c355f19d0f526f07aafe28c8b0c20cc0a303d85648b130062 WatchSource:0}: Error finding container 43bcb6471539c49c355f19d0f526f07aafe28c8b0c20cc0a303d85648b130062: Status 404 returned error can't find the container with id 43bcb6471539c49c355f19d0f526f07aafe28c8b0c20cc0a303d85648b130062
Nov 03 21:38:11.261769 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:11.261747    5244 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:38:11.831266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:11.831234    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk" event={"ID":"ea56e23a-3305-47a8-b1a1-51c222c2680e","Type":"ContainerStarted","Data":"43bcb6471539c49c355f19d0f526f07aafe28c8b0c20cc0a303d85648b130062"}
Nov 03 21:38:14.852616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:14.852580    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk" event={"ID":"ea56e23a-3305-47a8-b1a1-51c222c2680e","Type":"ContainerStarted","Data":"ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96"}
Nov 03 21:38:14.853033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:14.852711    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:14.966195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:14.966139    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk" podStartSLOduration=2.315866884 podStartE2EDuration="4.966122157s" totalImagesPullingTime="2.650255273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:14.965549841 +0000 UTC m=+1365.949546422" watchObservedRunningTime="2026-11-03 21:38:14.966122157 +0000 UTC m=+1365.950118717"
Nov 03 21:38:22.437820 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:38:22.437775    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:38:25.861731 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:25.861705    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:38:36.148357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.148286    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"]
Nov 03 21:38:36.156733 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.156706    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.206352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.203566    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.217167 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.204276    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sj7x\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.217557 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.217531    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.372354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.269280    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.372354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.367012    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.372354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.367080    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.372354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.367188    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.372354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.367239    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.372354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.367347    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.419217 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.381000    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5733b4-openshift-default-dockercfg-m5699\""
Nov 03 21:38:36.476777 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.461383    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"]
Nov 03 21:38:36.485912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.485748    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.486321 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.486210    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.486643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.486622    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.492915 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.492886    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.495238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.495210    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sj7x\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.496583 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.496558    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.501194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.491950    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.501194 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.500731    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.503193 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.501792    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.507361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.506914    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.507361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.507103    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.507361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.507249    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.511358 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.508486    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.514406 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.512209    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.569401 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.568696    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.571377 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.571348    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.591135 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.591054    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.593577 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.593397    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sj7x\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x\") pod \"e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") " pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:36.771621 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:36.771582    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:37.208826 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:37.208799    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"]
Nov 03 21:38:37.211309 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:37.211278    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e638c9_bb2f_4356_ba2b_3110b7000a9d.slice/crio-48382f15f0f517e2aa358c2194dc384562bf0da809a331a49d8232851ff1b7be WatchSource:0}: Error finding container 48382f15f0f517e2aa358c2194dc384562bf0da809a331a49d8232851ff1b7be: Status 404 returned error can't find the container with id 48382f15f0f517e2aa358c2194dc384562bf0da809a331a49d8232851ff1b7be
Nov 03 21:38:37.214017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:37.213981    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:38:37.214119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:37.214057    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:38:37.214119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:37.214090    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:38:37.999105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:37.999066    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" event={"ID":"b1e638c9-bb2f-4356-ba2b-3110b7000a9d","Type":"ContainerStarted","Data":"fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"}
Nov 03 21:38:37.999284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:37.999154    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" event={"ID":"b1e638c9-bb2f-4356-ba2b-3110b7000a9d","Type":"ContainerStarted","Data":"48382f15f0f517e2aa358c2194dc384562bf0da809a331a49d8232851ff1b7be"}
Nov 03 21:38:38.163718 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:38.163661    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podStartSLOduration=2.1636480909999998 podStartE2EDuration="2.163648091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:38.162760748 +0000 UTC m=+1389.146757347" watchObservedRunningTime="2026-11-03 21:38:38.163648091 +0000 UTC m=+1389.147644650"
Nov 03 21:38:38.773217 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:38.773171    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:38.779296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:38.779266    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:39.005480 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:39.005451    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:39.006413 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:39.006389    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:38:40.828120 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:40.828069    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:38:46.835282 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:46.835237    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"]
Nov 03 21:38:46.841053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:46.841016    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:46.878385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:46.878353    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"]
Nov 03 21:38:46.935698 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:46.935665    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5733b4\""
Nov 03 21:38:46.936198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:46.936173    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5733b4-dockercfg-vzdf6\""
Nov 03 21:38:47.023355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:46.987645    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.045367 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.043906    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltdj\" (UniqueName: \"kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj\") pod \"payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.080424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.080386    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"]
Nov 03 21:38:47.116419 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.108552    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"]
Nov 03 21:38:47.116419 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.108689    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.158742 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.148075    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.159196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.159164    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltdj\" (UniqueName: \"kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj\") pod \"payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.159619 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.159222    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.221107 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.221073    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltdj\" (UniqueName: \"kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj\") pod \"payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.261301 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.261259    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6k6\" (UniqueName: \"kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6\") pod \"payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.261552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.261317    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.363180 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.363147    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6k6\" (UniqueName: \"kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6\") pod \"payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.363436 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.363198    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.364025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.364007    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.420316 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.420241    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6k6\" (UniqueName: \"kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6\") pod \"payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") " pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.467342 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.467307    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:47.467558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.467468    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:47.850991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.850961    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"]
Nov 03 21:38:47.852644 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:47.852618    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485860b3_cec8_4522_b4f8_8058e8c1e6b0.slice/crio-e28c6ff15214262387575c323358a339f1d5c5f7d755dd8f42efa921969b9c5e WatchSource:0}: Error finding container e28c6ff15214262387575c323358a339f1d5c5f7d755dd8f42efa921969b9c5e: Status 404 returned error can't find the container with id e28c6ff15214262387575c323358a339f1d5c5f7d755dd8f42efa921969b9c5e
Nov 03 21:38:47.901837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:47.901811    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"]
Nov 03 21:38:47.915024 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:47.914989    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ae221b_a078_464f_b21e_1efbb8e2d229.slice/crio-5b46b1d5d1b1ae64bb2b183f4c73c642524f49ccea354e3d5d1e343ec2bf98de WatchSource:0}: Error finding container 5b46b1d5d1b1ae64bb2b183f4c73c642524f49ccea354e3d5d1e343ec2bf98de: Status 404 returned error can't find the container with id 5b46b1d5d1b1ae64bb2b183f4c73c642524f49ccea354e3d5d1e343ec2bf98de
Nov 03 21:38:48.205398 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.205356    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv" event={"ID":"01ae221b-a078-464f-b21e-1efbb8e2d229","Type":"ContainerStarted","Data":"a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e"}
Nov 03 21:38:48.205620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.205414    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv" event={"ID":"01ae221b-a078-464f-b21e-1efbb8e2d229","Type":"ContainerStarted","Data":"5b46b1d5d1b1ae64bb2b183f4c73c642524f49ccea354e3d5d1e343ec2bf98de"}
Nov 03 21:38:48.205620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.205466    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:48.207137 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.207101    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v" event={"ID":"485860b3-cec8-4522-b4f8-8058e8c1e6b0","Type":"ContainerStarted","Data":"3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e"}
Nov 03 21:38:48.207258 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.207161    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v" event={"ID":"485860b3-cec8-4522-b4f8-8058e8c1e6b0","Type":"ContainerStarted","Data":"e28c6ff15214262387575c323358a339f1d5c5f7d755dd8f42efa921969b9c5e"}
Nov 03 21:38:48.207258 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.207187    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:38:48.324199 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.324131    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv" podStartSLOduration=2.324112492 podStartE2EDuration="2.324112492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:48.320730484 +0000 UTC m=+1399.304727042" watchObservedRunningTime="2026-11-03 21:38:48.324112492 +0000 UTC m=+1399.308109052"
Nov 03 21:38:48.474411 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:48.467062    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v" podStartSLOduration=2.467039659 podStartE2EDuration="2.467039659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:48.451771369 +0000 UTC m=+1399.435767941" watchObservedRunningTime="2026-11-03 21:38:48.467039659 +0000 UTC m=+1399.451036219"
Nov 03 21:38:51.981422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:51.979264    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"]
Nov 03 21:38:52.083988 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.069750    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.115980 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.090015    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"]
Nov 03 21:38:52.154426 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.154399    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dad161\""
Nov 03 21:38:52.186343 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.186302    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") " pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.186545 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.186421    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvv2\" (UniqueName: \"kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2\") pod \"payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") " pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.203705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.203678    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dad161-dockercfg-t5mnn\""
Nov 03 21:38:52.287950 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.287845    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") " pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.287950 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.287893    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvv2\" (UniqueName: \"kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2\") pod \"payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") " pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.288634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.288611    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") " pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.344413 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.344381    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvv2\" (UniqueName: \"kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2\") pod \"payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") " pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.435663 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.435629    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:52.438160 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:38:52.438116    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:38:52.810909 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:52.810881    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"]
Nov 03 21:38:52.812734 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:52.812704    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610eb94f_50c1_4d1f_8a39_67b526733950.slice/crio-e38df239eb8e33a2e75c0db626757a7c8db90887460f14a5eb8a651763202b23 WatchSource:0}: Error finding container e38df239eb8e33a2e75c0db626757a7c8db90887460f14a5eb8a651763202b23: Status 404 returned error can't find the container with id e38df239eb8e33a2e75c0db626757a7c8db90887460f14a5eb8a651763202b23
Nov 03 21:38:53.241142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.241105    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk" event={"ID":"610eb94f-50c1-4d1f-8a39-67b526733950","Type":"ContainerStarted","Data":"5b7e04d9f046e52c06f2960c31f231f2fee8b20911a46d063fd885c90d75949a"}
Nov 03 21:38:53.241638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.241165    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk" event={"ID":"610eb94f-50c1-4d1f-8a39-67b526733950","Type":"ContainerStarted","Data":"e38df239eb8e33a2e75c0db626757a7c8db90887460f14a5eb8a651763202b23"}
Nov 03 21:38:53.241638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.241312    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:38:53.355045 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.354978    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk" podStartSLOduration=2.354956454 podStartE2EDuration="2.354956454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:53.352604925 +0000 UTC m=+1404.336601515" watchObservedRunningTime="2026-11-03 21:38:53.354956454 +0000 UTC m=+1404.338953015"
Nov 03 21:38:53.638772 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.638677    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7682-9a7469"]
Nov 03 21:38:53.643569 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.643547    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:38:53.695211 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.695181    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8dgqp\""
Nov 03 21:38:53.695421 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.695300    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-9a7469"]
Nov 03 21:38:53.744964 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.744920    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24ms\" (UniqueName: \"kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms\") pod \"test-curl-7682-9a7469\" (UID: \"027ac02b-9358-4aa3-8bc9-f0520c0c678d\") " pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:38:53.845873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.845836    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24ms\" (UniqueName: \"kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms\") pod \"test-curl-7682-9a7469\" (UID: \"027ac02b-9358-4aa3-8bc9-f0520c0c678d\") " pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:38:53.902220 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.902142    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24ms\" (UniqueName: \"kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms\") pod \"test-curl-7682-9a7469\" (UID: \"027ac02b-9358-4aa3-8bc9-f0520c0c678d\") " pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:38:53.957527 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:53.957486    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:38:54.331059 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:54.331025    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-9a7469"]
Nov 03 21:38:54.333023 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:38:54.332994    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027ac02b_9358_4aa3_8bc9_f0520c0c678d.slice/crio-d0dd1aa6966bd3b44e02d6ed8a7e02ca7da0b330b33ba6bec3e1a4b62f46c303 WatchSource:0}: Error finding container d0dd1aa6966bd3b44e02d6ed8a7e02ca7da0b330b33ba6bec3e1a4b62f46c303: Status 404 returned error can't find the container with id d0dd1aa6966bd3b44e02d6ed8a7e02ca7da0b330b33ba6bec3e1a4b62f46c303
Nov 03 21:38:55.255216 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:55.255183    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-9a7469" event={"ID":"027ac02b-9358-4aa3-8bc9-f0520c0c678d","Type":"ContainerStarted","Data":"d0dd1aa6966bd3b44e02d6ed8a7e02ca7da0b330b33ba6bec3e1a4b62f46c303"}
Nov 03 21:38:58.278434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:58.278393    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-9a7469" event={"ID":"027ac02b-9358-4aa3-8bc9-f0520c0c678d","Type":"ContainerStarted","Data":"a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a"}
Nov 03 21:38:58.389644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:58.389584    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7682-9a7469" podStartSLOduration=2.187325433 podStartE2EDuration="5.389565198s" totalImagesPullingTime="3.202239765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:38:58.388671734 +0000 UTC m=+1409.372668312" watchObservedRunningTime="2026-11-03 21:38:58.389565198 +0000 UTC m=+1409.373561757"
Nov 03 21:38:59.459364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:59.458552    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:38:59.459848 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:38:59.459426    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:39:00.132281 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:00.132240    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7682-9a7469"]
Nov 03 21:39:00.135925 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:00.135896    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7682-9a7469"]
Nov 03 21:39:00.476052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:00.475996    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7682-9a7469" podUID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" containerName="test-curl-7682-9a7469" containerID="cri-o://a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a" gracePeriod=2
Nov 03 21:39:00.526839 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:39:00.526799    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-9a7469\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:39:02.723117 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:02.723093    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:39:02.838806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:02.838707    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms\" (UniqueName: \"kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms\") pod \"027ac02b-9358-4aa3-8bc9-f0520c0c678d\" (UID: \"027ac02b-9358-4aa3-8bc9-f0520c0c678d\") "
Nov 03 21:39:02.840921 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:02.840895    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms" pod "027ac02b-9358-4aa3-8bc9-f0520c0c678d" (UID: "027ac02b-9358-4aa3-8bc9-f0520c0c678d"). InnerVolumeSpecName "kube-api-access-l24ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:39:02.940842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:02.940804    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24ms\" (UniqueName: \"kubernetes.io/projected/027ac02b-9358-4aa3-8bc9-f0520c0c678d-kube-api-access-l24ms\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:39:03.136626 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:03.136552    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" path="/var/lib/kubelet/pods/027ac02b-9358-4aa3-8bc9-f0520c0c678d/volumes"
Nov 03 21:39:03.495199 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:03.495161    5244 generic.go:356] "Generic (PLEG): container finished" podID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" containerID="a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a" exitCode=137
Nov 03 21:39:03.495199 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:03.495203    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-9a7469"
Nov 03 21:39:03.495480 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:03.495258    5244 scope.go:122] "RemoveContainer" containerID="a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a"
Nov 03 21:39:03.504178 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:03.504159    5244 scope.go:122] "RemoveContainer" containerID="a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a"
Nov 03 21:39:03.504474 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:39:03.504447    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a\": container with ID starting with a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a not found: ID does not exist" containerID="a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a"
Nov 03 21:39:03.504529 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:03.504483    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a"} err="failed to get container status \"a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a\": rpc error: code = NotFound desc = could not find container \"a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a\": container with ID starting with a61a3921464841d10e84dbd1508a7a0952bf09c86cac11a46c2dfe52455a5f1a not found: ID does not exist"
Nov 03 21:39:04.251811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:04.251787    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:39:07.781567 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.781523    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"]
Nov 03 21:39:07.783045 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.783013    5244 cpu_manager.go:414] "containerMap: removing container" podUID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" containerName="test-curl-7682-9a7469"
Nov 03 21:39:07.783208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.783194    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" containerName="test-curl-7682-9a7469"
Nov 03 21:39:07.783517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.783501    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="027ac02b-9358-4aa3-8bc9-f0520c0c678d" containerName="test-curl-7682-9a7469"
Nov 03 21:39:07.797618 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.789217    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:07.829530 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.829493    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"]
Nov 03 21:39:07.830163 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.830128    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffkk\" (UniqueName: \"kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk\") pod \"payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:07.830269 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.830184    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:07.876409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.876375    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fff189-dockercfg-qq7zs\""
Nov 03 21:39:07.915829 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.915797    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fff189\""
Nov 03 21:39:07.931409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.931379    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jffkk\" (UniqueName: \"kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk\") pod \"payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:07.931569 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.931419    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:07.932200 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.932175    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:07.989221 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:07.989180    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffkk\" (UniqueName: \"kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk\") pod \"payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:08.127662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:08.127564    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:08.503291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:08.503257    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"]
Nov 03 21:39:08.505358 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:39:08.505311    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bec8098_2187_4792_abba_d7cd6f8d3f6e.slice/crio-77fdc6e0ca8831eefaa9dbb3897d87aa45381da913a8e05538896a8479949ac7 WatchSource:0}: Error finding container 77fdc6e0ca8831eefaa9dbb3897d87aa45381da913a8e05538896a8479949ac7: Status 404 returned error can't find the container with id 77fdc6e0ca8831eefaa9dbb3897d87aa45381da913a8e05538896a8479949ac7
Nov 03 21:39:08.537033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:08.536997    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld" event={"ID":"0bec8098-2187-4792-abba-d7cd6f8d3f6e","Type":"ContainerStarted","Data":"77fdc6e0ca8831eefaa9dbb3897d87aa45381da913a8e05538896a8479949ac7"}
Nov 03 21:39:09.596170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:09.596125    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld" event={"ID":"0bec8098-2187-4792-abba-d7cd6f8d3f6e","Type":"ContainerStarted","Data":"69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd"}
Nov 03 21:39:09.602638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:09.602608    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:09.712146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:09.712081    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld" podStartSLOduration=2.71206439 podStartE2EDuration="2.71206439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:09.709458677 +0000 UTC m=+1420.693455248" watchObservedRunningTime="2026-11-03 21:39:09.71206439 +0000 UTC m=+1420.696060949"
Nov 03 21:39:20.605773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:20.605741    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:39:22.437176 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:39:22.437131    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:39:23.712680 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:23.712644    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:39:26.518344 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.518299    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"]
Nov 03 21:39:26.522595 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.522566    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.569097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.569064    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.569284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.569201    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.569284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.569251    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.569403 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.569288    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78ps\" (UniqueName: \"kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.576465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.576442    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5733b4-serving-cert\""
Nov 03 21:39:26.628623 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.628590    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"]
Nov 03 21:39:26.670266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.670232    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.670470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.670343    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.670470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.670393    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.670470 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:39:26.670417    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-serving-cert: secret "maas-api-e2e-api-b-5733b4-serving-cert" not found
Nov 03 21:39:26.670470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.670443    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78ps\" (UniqueName: \"kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.670681 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:39:26.670495    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls podName:16e69ac2-9171-4c52-8b73-54765d2f495e nodeName:}" failed. No retries permitted until 2026-11-03 21:39:27.170475491 +0000 UTC m=+1438.154472042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls") pod "maas-api-e2e-api-b-5733b4-84f74459-22kj6" (UID: "16e69ac2-9171-4c52-8b73-54765d2f495e") : secret "maas-api-e2e-api-b-5733b4-serving-cert" not found
Nov 03 21:39:26.671287 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.671263    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.673841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.673112    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:26.729561 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:26.729530    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78ps\" (UniqueName: \"kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:27.189434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:27.189397    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:27.194600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:27.192708    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\") pod \"maas-api-e2e-api-b-5733b4-84f74459-22kj6\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:27.454409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:27.444582    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:27.857398 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:27.852771    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"]
Nov 03 21:39:27.859413 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:39:27.859377    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e69ac2_9171_4c52_8b73_54765d2f495e.slice/crio-c6db4657f973b2bb5727095df2298cf3dd31eefc4f35a27085c1c2fc0199b9da WatchSource:0}: Error finding container c6db4657f973b2bb5727095df2298cf3dd31eefc4f35a27085c1c2fc0199b9da: Status 404 returned error can't find the container with id c6db4657f973b2bb5727095df2298cf3dd31eefc4f35a27085c1c2fc0199b9da
Nov 03 21:39:28.783370 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:28.781799    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6" event={"ID":"16e69ac2-9171-4c52-8b73-54765d2f495e","Type":"ContainerStarted","Data":"c6db4657f973b2bb5727095df2298cf3dd31eefc4f35a27085c1c2fc0199b9da"}
Nov 03 21:39:29.791063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:29.791032    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6" event={"ID":"16e69ac2-9171-4c52-8b73-54765d2f495e","Type":"ContainerStarted","Data":"4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf"}
Nov 03 21:39:29.791548 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:29.791512    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:29.913158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:29.913107    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6" podStartSLOduration=2.988639413 podStartE2EDuration="3.913092091s" totalImagesPullingTime="924.452678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:29.909364433 +0000 UTC m=+1440.893361015" watchObservedRunningTime="2026-11-03 21:39:29.913092091 +0000 UTC m=+1440.897088654"
Nov 03 21:39:34.821445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:34.821308    5244 generic.go:356] "Generic (PLEG): container finished" podID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerID="2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b" exitCode=0
Nov 03 21:39:34.821445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:34.821386    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" event={"ID":"ae5e5831-49d4-4fc9-adf7-73177d082a13","Type":"ContainerDied","Data":"2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"}
Nov 03 21:39:34.822049 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:34.821972    5244 scope.go:122] "RemoveContainer" containerID="2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"
Nov 03 21:39:34.823679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:34.823651    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:39:34.823765 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:34.823720    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:39:34.823765 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:34.823755    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:39:35.797500 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:35.797458    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:39:35.802846 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:35.802824    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:39:35.829135 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:35.829100    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" event={"ID":"ae5e5831-49d4-4fc9-adf7-73177d082a13","Type":"ContainerStarted","Data":"7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779"}
Nov 03 21:39:36.100280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:36.100199    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:39:36.101594 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:36.101570    5244 patch_prober.go:29] interesting pod/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 21:39:36.101722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:36.101620    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 21:39:37.099815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:37.099734    5244 patch_prober.go:29] interesting pod/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 21:39:37.099815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:37.099789    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 21:39:38.100715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:38.100676    5244 patch_prober.go:29] interesting pod/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 21:39:38.101165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:38.100752    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 21:39:39.099998 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:39.099960    5244 patch_prober.go:29] interesting pod/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 21:39:39.100188 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:39.100027    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 21:39:40.100046 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:40.100007    5244 patch_prober.go:29] interesting pod/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 21:39:40.100470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:40.100074    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 21:39:41.104830 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:41.104797    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:39:41.105227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:41.105069    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:39:41.105782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:39:41.105763    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:39:52.461065 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:39:52.461008    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:40:00.048881 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:00.048851    5244 generic.go:356] "Generic (PLEG): container finished" podID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerID="fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a" exitCode=0
Nov 03 21:40:00.049449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:00.048932    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" event={"ID":"b1e638c9-bb2f-4356-ba2b-3110b7000a9d","Type":"ContainerDied","Data":"fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"}
Nov 03 21:40:00.049525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:00.049499    5244 scope.go:122] "RemoveContainer" containerID="fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"
Nov 03 21:40:00.051056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:00.051021    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:40:00.051166 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:00.051130    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:40:00.051228 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:00.051182    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:40:01.064076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.064037    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" event={"ID":"b1e638c9-bb2f-4356-ba2b-3110b7000a9d","Type":"ContainerStarted","Data":"94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd"}
Nov 03 21:40:01.773425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.773387    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:40:01.773661 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.773639    5244 patch_prober.go:29] interesting pod/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:40:01.773707 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.773689    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:40:01.857945 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.857904    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"]
Nov 03 21:40:01.863034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.863010    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:01.899380 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.899349    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"]
Nov 03 21:40:01.914971 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.914929    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fff189-serving-cert\""
Nov 03 21:40:01.996712 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.996677    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:01.996912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.996728    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497tb\" (UniqueName: \"kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:01.996912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.996897    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:01.997054 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:01.996958    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.106538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.104347    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497tb\" (UniqueName: \"kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.106538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.104493    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.106538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.104596    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.106538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.104837    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.106538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.105988    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.128815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.128010    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.147425 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.143428    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.193354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.189806    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-497tb\" (UniqueName: \"kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb\") pod \"maas-api-e2e-api-a-fff189-564c5496f4-5l4t8\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.477033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.476993    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:02.773274 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.773183    5244 patch_prober.go:29] interesting pod/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:40:02.773274 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.773239    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:40:02.876525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:02.876499    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"]
Nov 03 21:40:02.878119 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:40:02.878089    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2377367_0055_4a96_84b6_b96f77a3ea32.slice/crio-5dde3667c8961f8f40b6c7191fcaffffd6429810d6fdd9d5eee1d54d28fb1621 WatchSource:0}: Error finding container 5dde3667c8961f8f40b6c7191fcaffffd6429810d6fdd9d5eee1d54d28fb1621: Status 404 returned error can't find the container with id 5dde3667c8961f8f40b6c7191fcaffffd6429810d6fdd9d5eee1d54d28fb1621
Nov 03 21:40:03.077055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:03.076965    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8" event={"ID":"d2377367-0055-4a96-84b6-b96f77a3ea32","Type":"ContainerStarted","Data":"5dde3667c8961f8f40b6c7191fcaffffd6429810d6fdd9d5eee1d54d28fb1621"}
Nov 03 21:40:03.773063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:03.773029    5244 patch_prober.go:29] interesting pod/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:40:03.773482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:03.773083    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:40:04.085190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:04.085080    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8" event={"ID":"d2377367-0055-4a96-84b6-b96f77a3ea32","Type":"ContainerStarted","Data":"0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103"}
Nov 03 21:40:04.085406 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:04.085286    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:04.779357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:04.777780    5244 patch_prober.go:29] interesting pod/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:40:04.779357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:04.777860    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:40:05.772996 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:05.772961    5244 patch_prober.go:29] interesting pod/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 21:40:05.773282 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:05.773017    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 21:40:06.792490 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:06.792454    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:40:06.793524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:06.793493    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:40:06.795405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:06.795368    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:40:06.961046 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:06.960990    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8" podStartSLOduration=5.121945276 podStartE2EDuration="5.960975923s" totalImagesPullingTime="839.030647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:04.232550213 +0000 UTC m=+1475.216546772" watchObservedRunningTime="2026-11-03 21:40:06.960975923 +0000 UTC m=+1477.944972494"
Nov 03 21:40:09.888041 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:09.888004    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:40:10.147289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:10.147211    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:40:22.438649 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:40:22.438600    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:40:52.437673 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:40:52.437633    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:40:54.619576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:40:54.619540    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:41:03.799974 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.799927    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"]
Nov 03 21:41:03.804588 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.804560    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:03.908165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.907741    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-85d7ff-serving-cert\""
Nov 03 21:41:03.912104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.912073    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"]
Nov 03 21:41:03.958506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.958471    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:03.958782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.958755    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplsb\" (UniqueName: \"kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:03.958948 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.958919    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:03.959059 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:03.959040    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.060817 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.060720    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.060817 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.060805    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.061066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.060861    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplsb\" (UniqueName: \"kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.061066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.060942    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.061205 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:41:04.061104    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-serving-cert: secret "maas-api-e2e-api-b-85d7ff-serving-cert" not found
Nov 03 21:41:04.061205 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:41:04.061192    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls podName:961af80d-53aa-43f4-b164-d331e017aea4 nodeName:}" failed. No retries permitted until 2026-11-03 21:41:04.56117376 +0000 UTC m=+1535.545170296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls") pod "maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" (UID: "961af80d-53aa-43f4-b164-d331e017aea4") : secret "maas-api-e2e-api-b-85d7ff-serving-cert" not found
Nov 03 21:41:04.061582 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.061560    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.063521 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.063503    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.135360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.131999    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplsb\" (UniqueName: \"kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.577797 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.577764    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.580368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.580340    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\") pod \"maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:04.718517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:04.718465    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:05.106374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:05.106289    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"]
Nov 03 21:41:05.107894 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:41:05.107864    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961af80d_53aa_43f4_b164_d331e017aea4.slice/crio-1c5c4a8a4d35dd1ce8996d9dd9e86031aeab287e18e286ed2c3e0b904493d849 WatchSource:0}: Error finding container 1c5c4a8a4d35dd1ce8996d9dd9e86031aeab287e18e286ed2c3e0b904493d849: Status 404 returned error can't find the container with id 1c5c4a8a4d35dd1ce8996d9dd9e86031aeab287e18e286ed2c3e0b904493d849
Nov 03 21:41:05.763265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:05.763222    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" event={"ID":"961af80d-53aa-43f4-b164-d331e017aea4","Type":"ContainerStarted","Data":"1c5c4a8a4d35dd1ce8996d9dd9e86031aeab287e18e286ed2c3e0b904493d849"}
Nov 03 21:41:06.312615 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:06.312527    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"]
Nov 03 21:41:06.313003 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:06.312721    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy" containerID="cri-o://152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750" gracePeriod=30
Nov 03 21:41:06.770591 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:06.770545    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" event={"ID":"961af80d-53aa-43f4-b164-d331e017aea4","Type":"ContainerStarted","Data":"1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726"}
Nov 03 21:41:06.770776 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:06.770716    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:06.905826 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:06.905712    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" podStartSLOduration=2.970173979 podStartE2EDuration="3.905664252s" totalImagesPullingTime="935.490273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:06.891619599 +0000 UTC m=+1537.875616159" watchObservedRunningTime="2026-11-03 21:41:06.905664252 +0000 UTC m=+1537.889660862"
Nov 03 21:41:11.578744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.578716    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:41:11.676754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676647    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.676754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676709    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.676754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676754    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.677067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676781    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.677067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676810    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.677067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676942    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.677067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.676960    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.677067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677051    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677060    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677100    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677130    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert\") pod \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\" (UID: \"adfb5a45-0f96-4bd5-beb9-cec9e15c2171\") "
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677138    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677236    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677531    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677720    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677733    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677743    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677755    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.678017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.677768    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.679271 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.679245    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:41:11.679454 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.679430    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:41:11.679603 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.679583    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "kube-api-access-487mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.679819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.679791    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token" pod "adfb5a45-0f96-4bd5-beb9-cec9e15c2171" (UID: "adfb5a45-0f96-4bd5-beb9-cec9e15c2171"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:41:11.779275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.779230    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.779275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.779274    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487mk\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-kube-api-access-487mk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.779535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.779291    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.779535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.779304    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfb5a45-0f96-4bd5-beb9-cec9e15c2171-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:41:11.805540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.805504    5244 generic.go:356] "Generic (PLEG): container finished" podID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerID="152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750" exitCode=0
Nov 03 21:41:11.805730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.805569    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"
Nov 03 21:41:11.805730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.805599    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" event={"ID":"adfb5a45-0f96-4bd5-beb9-cec9e15c2171","Type":"ContainerDied","Data":"152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750"}
Nov 03 21:41:11.805730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.805665    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq" event={"ID":"adfb5a45-0f96-4bd5-beb9-cec9e15c2171","Type":"ContainerDied","Data":"87f84d65d4dae73f5850170dd6d6778277ad8eed88dd82d5d645eb0867d58039"}
Nov 03 21:41:11.805730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.805698    5244 scope.go:122] "RemoveContainer" containerID="152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750"
Nov 03 21:41:11.832985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.832955    5244 scope.go:122] "RemoveContainer" containerID="152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750"
Nov 03 21:41:11.833412 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:41:11.833383    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750\": container with ID starting with 152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750 not found: ID does not exist" containerID="152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750"
Nov 03 21:41:11.833541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.833418    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750"} err="failed to get container status \"152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750\": rpc error: code = NotFound desc = could not find container \"152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750\": container with ID starting with 152eba57d020c2c139e305b2dce33dc1688cf846a2e7a695cc0f09a0d7601750 not found: ID does not exist"
Nov 03 21:41:11.970093 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.970048    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"]
Nov 03 21:41:11.975187 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:11.975158    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-kmxpq"]
Nov 03 21:41:12.784422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:12.784395    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:41:13.157408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:13.152961    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" path="/var/lib/kubelet/pods/adfb5a45-0f96-4bd5-beb9-cec9e15c2171/volumes"
Nov 03 21:41:22.535462 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:41:22.535408    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:41:31.477581 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:41:31.477546    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:41:52.436846 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:41:52.436808    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:42:15.438649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:15.438604    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"]
Nov 03 21:42:15.439195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:15.438811    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy" containerID="cri-o://37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb" gracePeriod=30
Nov 03 21:42:17.392485 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:17.392452    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"]
Nov 03 21:42:17.392924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:17.392661    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" containerID="cri-o://5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9" gracePeriod=30
Nov 03 21:42:17.664678 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:17.664581    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:42:20.717344 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.717310    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:42:20.855039 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.854958    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855047    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855116    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855164    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855210    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855258    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855317    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855498 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855469    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855561 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855541    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:20.855628 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855553    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:20.855628 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855556    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:20.855729 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855664    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token\") pod \"30365623-40de-4ff3-937c-ecea7cb4f62d\" (UID: \"30365623-40de-4ff3-937c-ecea7cb4f62d\") "
Nov 03 21:42:20.855829 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855809    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:20.856061 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.855878    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:20.856360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.856311    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30365623-40de-4ff3-937c-ecea7cb4f62d-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.856553 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.856362    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.856553 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.856378    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.856553 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.856392    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.856553 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.856406    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.857715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.857675    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:20.857834 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.857792    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:20.858149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.858123    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "kube-api-access-wj5g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:20.858191 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.858183    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token" pod "30365623-40de-4ff3-937c-ecea7cb4f62d" (UID: "30365623-40de-4ff3-937c-ecea7cb4f62d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:20.957720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.957684    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5g8\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-kube-api-access-wj5g8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.957720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.957716    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.957720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.957726    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:20.958016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:20.957734    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30365623-40de-4ff3-937c-ecea7cb4f62d-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:21.492814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.492773    5244 generic.go:356] "Generic (PLEG): container finished" podID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerID="37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb" exitCode=0
Nov 03 21:42:21.492992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.492820    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"
Nov 03 21:42:21.492992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.492856    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" event={"ID":"30365623-40de-4ff3-937c-ecea7cb4f62d","Type":"ContainerDied","Data":"37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb"}
Nov 03 21:42:21.492992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.492911    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7" event={"ID":"30365623-40de-4ff3-937c-ecea7cb4f62d","Type":"ContainerDied","Data":"c1e3702eb32c37c41c868263badd385dd0eeb2a4540f06c2d18dc589783ab654"}
Nov 03 21:42:21.492992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.492934    5244 scope.go:122] "RemoveContainer" containerID="37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb"
Nov 03 21:42:21.515366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.515320    5244 scope.go:122] "RemoveContainer" containerID="37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb"
Nov 03 21:42:21.515792 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:21.515763    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb\": container with ID starting with 37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb not found: ID does not exist" containerID="37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb"
Nov 03 21:42:21.515892 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.515810    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb"} err="failed to get container status \"37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb\": rpc error: code = NotFound desc = could not find container \"37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb\": container with ID starting with 37df2cdacebf41d58974dbfb190757008272cd853f136ee74dd874aeee2baefb not found: ID does not exist"
Nov 03 21:42:21.655630 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.655594    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"]
Nov 03 21:42:21.705241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:21.705205    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-kt4l7"]
Nov 03 21:42:22.364890 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.364849    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:42:22.365297 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.364913    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:42:22.437497 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:22.437460    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:42:22.658895 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.658871    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:42:22.781518 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781483    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781619    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781671    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781731    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781785    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781817    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781855    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781880    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.781991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781898    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.781991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.781917    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs\") pod \"8a80c109-0e27-45d0-bb6d-a04678c1586e\" (UID: \"8a80c109-0e27-45d0-bb6d-a04678c1586e\") "
Nov 03 21:42:22.782181 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.782067    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.782952 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.782558    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.782952 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.782586    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.782952 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.782619    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.783233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.783048    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:22.783408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.783368    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.784560 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.784524    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "kube-api-access-ph646". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:22.784674 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.784625    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:22.784674 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.784662    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:22.785875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.785842    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy" pod "8a80c109-0e27-45d0-bb6d-a04678c1586e" (UID: "8a80c109-0e27-45d0-bb6d-a04678c1586e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.883933    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.883969    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.883978    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80c109-0e27-45d0-bb6d-a04678c1586e-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.883986    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.883996    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph646\" (UniqueName: \"kubernetes.io/projected/8a80c109-0e27-45d0-bb6d-a04678c1586e-kube-api-access-ph646\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.884005    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80c109-0e27-45d0-bb6d-a04678c1586e-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:22.884017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:22.884015    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80c109-0e27-45d0-bb6d-a04678c1586e-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:23.137955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.137862    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" path="/var/lib/kubelet/pods/30365623-40de-4ff3-937c-ecea7cb4f62d/volumes"
Nov 03 21:42:23.508214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.508181    5244 generic.go:356] "Generic (PLEG): container finished" podID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerID="5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9" exitCode=0
Nov 03 21:42:23.508773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.508234    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"
Nov 03 21:42:23.508773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.508254    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" event={"ID":"8a80c109-0e27-45d0-bb6d-a04678c1586e","Type":"ContainerDied","Data":"5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9"}
Nov 03 21:42:23.508773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.508308    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc" event={"ID":"8a80c109-0e27-45d0-bb6d-a04678c1586e","Type":"ContainerDied","Data":"80967f4e0f8edcb88212c186ce3474f93e5192fafd40ae79d92d646a44b454f4"}
Nov 03 21:42:23.508773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.508389    5244 scope.go:122] "RemoveContainer" containerID="5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9"
Nov 03 21:42:23.519440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.519418    5244 scope.go:122] "RemoveContainer" containerID="5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9"
Nov 03 21:42:23.519763 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:23.519736    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9\": container with ID starting with 5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9 not found: ID does not exist" containerID="5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9"
Nov 03 21:42:23.519865 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.519775    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9"} err="failed to get container status \"5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9\": rpc error: code = NotFound desc = could not find container \"5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9\": container with ID starting with 5a1c79f6153ae3a3ba69cf5e4181735810c279cad316f5896957dad7498472a9 not found: ID does not exist"
Nov 03 21:42:23.721686 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.721649    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"]
Nov 03 21:42:23.771404 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:23.771296    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-r85cc"]
Nov 03 21:42:25.136494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:25.136460    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" path="/var/lib/kubelet/pods/8a80c109-0e27-45d0-bb6d-a04678c1586e/volumes"
Nov 03 21:42:31.733960 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:31.733923    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"]
Nov 03 21:42:34.393103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393063    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"]
Nov 03 21:42:34.393730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393713    5244 cpu_manager.go:414] "containerMap: removing container" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy"
Nov 03 21:42:34.393778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393731    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy"
Nov 03 21:42:34.393778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393746    5244 cpu_manager.go:414] "containerMap: removing container" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy"
Nov 03 21:42:34.393778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393751    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy"
Nov 03 21:42:34.393870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393781    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy"
Nov 03 21:42:34.393870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393788    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy"
Nov 03 21:42:34.393933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393874    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="30365623-40de-4ff3-937c-ecea7cb4f62d" containerName="istio-proxy"
Nov 03 21:42:34.393933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393885    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfb5a45-0f96-4bd5-beb9-cec9e15c2171" containerName="istio-proxy"
Nov 03 21:42:34.393933 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.393894    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a80c109-0e27-45d0-bb6d-a04678c1586e" containerName="istio-proxy"
Nov 03 21:42:34.398855 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.398836    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.495026 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.494988    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"]
Nov 03 21:42:34.512539 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512502    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.512738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512560    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7w6\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.512738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512608    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.512738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512645    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.512738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512690    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.512738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512719    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.513027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.512905    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.513079 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.513037    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.513323 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.513302    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614234 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614188    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614249    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7w6\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614293    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614350    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614400    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614427    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614471    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614556    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.614814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.614609    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.615231 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.615204    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.615538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.615514    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.615844 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.615795    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.616053 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.616031    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.616706 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.616684    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.617484 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.617459    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.619851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.619826    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.672270 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.672203    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.672270 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.672235    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7w6\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:34.712593 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:34.712548    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:35.142789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.142763    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"]
Nov 03 21:42:35.144557 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:42:35.144530    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc93a84_e1d3_479f_bee2_26843f2214f6.slice/crio-75536710aec30f0828cdfdb7a1a367d3ef599841d00231a54c8c3f54e6911f7e WatchSource:0}: Error finding container 75536710aec30f0828cdfdb7a1a367d3ef599841d00231a54c8c3f54e6911f7e: Status 404 returned error can't find the container with id 75536710aec30f0828cdfdb7a1a367d3ef599841d00231a54c8c3f54e6911f7e
Nov 03 21:42:35.147242 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.147210    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:42:35.147367 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.147275    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:42:35.147367 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.147304    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:42:35.639227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.637827    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" event={"ID":"acc93a84-e1d3-479f-bee2-26843f2214f6","Type":"ContainerStarted","Data":"c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6"}
Nov 03 21:42:35.639227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.637894    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" event={"ID":"acc93a84-e1d3-479f-bee2-26843f2214f6","Type":"ContainerStarted","Data":"75536710aec30f0828cdfdb7a1a367d3ef599841d00231a54c8c3f54e6911f7e"}
Nov 03 21:42:35.799748 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:35.775121    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:36.714318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:36.714231    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:42:36.714727 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:36.714314    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 21:42:36.749721 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:36.749639    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk" podUID="ea56e23a-3305-47a8-b1a1-51c222c2680e" containerName="payload-processing" containerID="cri-o://ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96" gracePeriod=25
Nov 03 21:42:37.071467 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.071441    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:42:37.184423 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.184365    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podStartSLOduration=3.184318864 podStartE2EDuration="3.184318864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:35.774744405 +0000 UTC m=+1626.758740964" watchObservedRunningTime="2026-11-03 21:42:37.184318864 +0000 UTC m=+1628.168315480"
Nov 03 21:42:37.204802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.204770    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl\" (UniqueName: \"kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl\") pod \"ea56e23a-3305-47a8-b1a1-51c222c2680e\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") "
Nov 03 21:42:37.204995 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.204933    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume\") pod \"ea56e23a-3305-47a8-b1a1-51c222c2680e\" (UID: \"ea56e23a-3305-47a8-b1a1-51c222c2680e\") "
Nov 03 21:42:37.205408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.205376    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume" pod "ea56e23a-3305-47a8-b1a1-51c222c2680e" (UID: "ea56e23a-3305-47a8-b1a1-51c222c2680e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:37.208338 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.208299    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl" pod "ea56e23a-3305-47a8-b1a1-51c222c2680e" (UID: "ea56e23a-3305-47a8-b1a1-51c222c2680e"). InnerVolumeSpecName "kube-api-access-lxgxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:37.306819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.306729    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxgxl\" (UniqueName: \"kubernetes.io/projected/ea56e23a-3305-47a8-b1a1-51c222c2680e-kube-api-access-lxgxl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:37.306819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.306765    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea56e23a-3305-47a8-b1a1-51c222c2680e-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:37.666186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.666104    5244 generic.go:356] "Generic (PLEG): container finished" podID="ea56e23a-3305-47a8-b1a1-51c222c2680e" containerID="ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96" exitCode=0
Nov 03 21:42:37.666479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.666460    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"
Nov 03 21:42:37.666563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.666530    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk" event={"ID":"ea56e23a-3305-47a8-b1a1-51c222c2680e","Type":"ContainerDied","Data":"ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96"}
Nov 03 21:42:37.666630 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.666617    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk" event={"ID":"ea56e23a-3305-47a8-b1a1-51c222c2680e","Type":"ContainerDied","Data":"43bcb6471539c49c355f19d0f526f07aafe28c8b0c20cc0a303d85648b130062"}
Nov 03 21:42:37.666669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.666644    5244 scope.go:122] "RemoveContainer" containerID="ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96"
Nov 03 21:42:37.679226 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.679207    5244 scope.go:122] "RemoveContainer" containerID="ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96"
Nov 03 21:42:37.679557 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:37.679534    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96\": container with ID starting with ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96 not found: ID does not exist" containerID="ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96"
Nov 03 21:42:37.679639 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.679567    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96"} err="failed to get container status \"ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96\": rpc error: code = NotFound desc = could not find container \"ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96\": container with ID starting with ee2785b285fd40fc88780b42dc2027b061dec8ca8382542de2bdf3a4811b1b96 not found: ID does not exist"
Nov 03 21:42:37.713052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.713020    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:42:37.713222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.713073    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 21:42:37.853802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.853730    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"]
Nov 03 21:42:37.863484 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.863445    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30f143-d69dcf544-4pnqk"]
Nov 03 21:42:37.882457 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.882418    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"]
Nov 03 21:42:37.882698 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.882673    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" containerID="cri-o://02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75" gracePeriod=30
Nov 03 21:42:37.893681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.893640    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"]
Nov 03 21:42:37.893964 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:37.893928    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" containerID="cri-o://6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397" gracePeriod=30
Nov 03 21:42:38.713627 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:38.713595    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:42:38.713808 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:38.713647    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 21:42:39.159984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:39.159901    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea56e23a-3305-47a8-b1a1-51c222c2680e" path="/var/lib/kubelet/pods/ea56e23a-3305-47a8-b1a1-51c222c2680e/volumes"
Nov 03 21:42:39.713915 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:39.713878    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:42:39.714108 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:39.713954    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 21:42:39.818794 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:39.818753    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"]
Nov 03 21:42:39.950542 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:39.950507    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"]
Nov 03 21:42:40.734358 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:40.734158    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:42:40.734358 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:40.734240    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 21:42:41.713833 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:41.713788    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 21:42:41.714156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:41.714110    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 21:42:42.740076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:42.740045    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:42:42.740495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:42.740110    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:42:42.814129 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:42.814098    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:42.814452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:42.814432    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:42.815168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:42.815153    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:42:43.118815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.118473    5244 patch_prober.go:29] interesting pod/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 21:42:43.118815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.118537    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 21:42:43.330870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.330843    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:42:43.334098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.334076    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:42:43.399996 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.399964    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400078    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400118    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400158    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400191    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400237    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400291    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400299    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400343    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400350    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.400407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400385    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400419    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400479    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400528    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400536    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400617    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400647    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400678    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:43.400755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400683    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400756    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400771    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400804    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400832    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket\") pod \"ad382be0-f946-4823-855a-1bed3a5e9c84\" (UID: \"ad382be0-f946-4823-855a-1bed3a5e9c84\") "
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400865    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token\") pod \"01945076-86c3-42f5-bf87-8cfbe2950a36\" (UID: \"01945076-86c3-42f5-bf87-8cfbe2950a36\") "
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400878    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.400912    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401009    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.401357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401193    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401666    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401688    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401704    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401717    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401731    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01945076-86c3-42f5-bf87-8cfbe2950a36-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401744    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401757    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401771    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad382be0-f946-4823-855a-1bed3a5e9c84-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401786    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.401946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.401799    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.403228 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.403184    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.403399 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.403303    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:43.403611 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.403584    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "kube-api-access-rdjnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:43.403812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.403792    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "kube-api-access-tlnhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:43.403889 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.403814    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:43.404161 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.404145    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:43.404770 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.404738    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy" pod "ad382be0-f946-4823-855a-1bed3a5e9c84" (UID: "ad382be0-f946-4823-855a-1bed3a5e9c84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:42:43.405320 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.405294    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo" pod "01945076-86c3-42f5-bf87-8cfbe2950a36" (UID: "01945076-86c3-42f5-bf87-8cfbe2950a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:42:43.502642 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502607    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502642 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502635    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502642 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502645    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdjnd\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-kube-api-access-rdjnd\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502654    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502672    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlnhh\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-kube-api-access-tlnhh\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502683    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502693    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad382be0-f946-4823-855a-1bed3a5e9c84-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.502878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.502731    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01945076-86c3-42f5-bf87-8cfbe2950a36-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:43.775602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.775563    5244 generic.go:356] "Generic (PLEG): container finished" podID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerID="02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75" exitCode=0
Nov 03 21:42:43.776049 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.775612    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"
Nov 03 21:42:43.776049 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.775639    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" event={"ID":"ad382be0-f946-4823-855a-1bed3a5e9c84","Type":"ContainerDied","Data":"02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75"}
Nov 03 21:42:43.776049 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.775697    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l" event={"ID":"ad382be0-f946-4823-855a-1bed3a5e9c84","Type":"ContainerDied","Data":"49f3928c503662c4e9839846ead3ad48e8f09112c85120d029afd67eb99a40e7"}
Nov 03 21:42:43.776049 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.775723    5244 scope.go:122] "RemoveContainer" containerID="02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75"
Nov 03 21:42:43.777492 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.777467    5244 generic.go:356] "Generic (PLEG): container finished" podID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerID="6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397" exitCode=0
Nov 03 21:42:43.777638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.777611    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" event={"ID":"01945076-86c3-42f5-bf87-8cfbe2950a36","Type":"ContainerDied","Data":"6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397"}
Nov 03 21:42:43.777686 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.777676    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"
Nov 03 21:42:43.777724 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.777680    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t" event={"ID":"01945076-86c3-42f5-bf87-8cfbe2950a36","Type":"ContainerDied","Data":"72c163ad30e1cf6fc473227323ba2eb22a335ee9f348092e7a00935048bab4cf"}
Nov 03 21:42:43.788692 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.788617    5244 scope.go:122] "RemoveContainer" containerID="02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75"
Nov 03 21:42:43.789006 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:43.788981    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75\": container with ID starting with 02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75 not found: ID does not exist" containerID="02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75"
Nov 03 21:42:43.789083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.789019    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75"} err="failed to get container status \"02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75\": rpc error: code = NotFound desc = could not find container \"02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75\": container with ID starting with 02fcea7f83d6316ff0b1ebbd480e7bb9676d9a94d88c79a542b09638c52e0c75 not found: ID does not exist"
Nov 03 21:42:43.789083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.789042    5244 scope.go:122] "RemoveContainer" containerID="6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397"
Nov 03 21:42:43.798745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.798722    5244 scope.go:122] "RemoveContainer" containerID="6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397"
Nov 03 21:42:43.799051 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:43.799032    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397\": container with ID starting with 6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397 not found: ID does not exist" containerID="6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397"
Nov 03 21:42:43.799115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.799060    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397"} err="failed to get container status \"6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397\": rpc error: code = NotFound desc = could not find container \"6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397\": container with ID starting with 6662f1be4194eb397631c67dfe78da6a4202dc7274efa80685a1056633e2a397 not found: ID does not exist"
Nov 03 21:42:43.959397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.945858    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"]
Nov 03 21:42:43.989360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:43.989299    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-8vg7l"]
Nov 03 21:42:44.097497 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:44.097406    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"]
Nov 03 21:42:44.101163 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:44.101135    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-da5f0b-openshift-default-bdb546fdc-pt65t"]
Nov 03 21:42:44.819541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:44.819479    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v" podUID="485860b3-cec8-4522-b4f8-8058e8c1e6b0" containerName="payload-processing" containerID="cri-o://3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e" gracePeriod=25
Nov 03 21:42:44.951505 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:44.951463    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv" podUID="01ae221b-a078-464f-b21e-1efbb8e2d229" containerName="payload-pre-processing" containerID="cri-o://a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e" gracePeriod=25
Nov 03 21:42:45.092717 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.092688    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:42:45.136732 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.135319    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6\" (UniqueName: \"kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6\") pod \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") "
Nov 03 21:42:45.136732 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.135459    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume\") pod \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\" (UID: \"485860b3-cec8-4522-b4f8-8058e8c1e6b0\") "
Nov 03 21:42:45.136732 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.136504    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume" pod "485860b3-cec8-4522-b4f8-8058e8c1e6b0" (UID: "485860b3-cec8-4522-b4f8-8058e8c1e6b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:45.138972 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.138942    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6" pod "485860b3-cec8-4522-b4f8-8058e8c1e6b0" (UID: "485860b3-cec8-4522-b4f8-8058e8c1e6b0"). InnerVolumeSpecName "kube-api-access-zg6k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:45.152280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.152241    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" path="/var/lib/kubelet/pods/01945076-86c3-42f5-bf87-8cfbe2950a36/volumes"
Nov 03 21:42:45.153077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.153051    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" path="/var/lib/kubelet/pods/ad382be0-f946-4823-855a-1bed3a5e9c84/volumes"
Nov 03 21:42:45.212462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.212436    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:42:45.238247 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.238206    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6k6\" (UniqueName: \"kubernetes.io/projected/485860b3-cec8-4522-b4f8-8058e8c1e6b0-kube-api-access-zg6k6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:45.238247 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.238238    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485860b3-cec8-4522-b4f8-8058e8c1e6b0-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:45.339477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.339299    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume\") pod \"01ae221b-a078-464f-b21e-1efbb8e2d229\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") "
Nov 03 21:42:45.339477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.339461    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj\" (UniqueName: \"kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj\") pod \"01ae221b-a078-464f-b21e-1efbb8e2d229\" (UID: \"01ae221b-a078-464f-b21e-1efbb8e2d229\") "
Nov 03 21:42:45.339767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.339740    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume" pod "01ae221b-a078-464f-b21e-1efbb8e2d229" (UID: "01ae221b-a078-464f-b21e-1efbb8e2d229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:42:45.340077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.340056    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ae221b-a078-464f-b21e-1efbb8e2d229-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:45.341870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.341841    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj" pod "01ae221b-a078-464f-b21e-1efbb8e2d229" (UID: "01ae221b-a078-464f-b21e-1efbb8e2d229"). InnerVolumeSpecName "kube-api-access-qltdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:42:45.441799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.441763    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltdj\" (UniqueName: \"kubernetes.io/projected/01ae221b-a078-464f-b21e-1efbb8e2d229-kube-api-access-qltdj\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:42:45.800789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.800750    5244 generic.go:356] "Generic (PLEG): container finished" podID="01ae221b-a078-464f-b21e-1efbb8e2d229" containerID="a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e" exitCode=0
Nov 03 21:42:45.800994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.800807    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"
Nov 03 21:42:45.800994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.800843    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv" event={"ID":"01ae221b-a078-464f-b21e-1efbb8e2d229","Type":"ContainerDied","Data":"a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e"}
Nov 03 21:42:45.800994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.800898    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv" event={"ID":"01ae221b-a078-464f-b21e-1efbb8e2d229","Type":"ContainerDied","Data":"5b46b1d5d1b1ae64bb2b183f4c73c642524f49ccea354e3d5d1e343ec2bf98de"}
Nov 03 21:42:45.800994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.800928    5244 scope.go:122] "RemoveContainer" containerID="a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e"
Nov 03 21:42:45.802528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.802503    5244 generic.go:356] "Generic (PLEG): container finished" podID="485860b3-cec8-4522-b4f8-8058e8c1e6b0" containerID="3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e" exitCode=0
Nov 03 21:42:45.802649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.802539    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v" event={"ID":"485860b3-cec8-4522-b4f8-8058e8c1e6b0","Type":"ContainerDied","Data":"3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e"}
Nov 03 21:42:45.802649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.802580    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v" event={"ID":"485860b3-cec8-4522-b4f8-8058e8c1e6b0","Type":"ContainerDied","Data":"e28c6ff15214262387575c323358a339f1d5c5f7d755dd8f42efa921969b9c5e"}
Nov 03 21:42:45.802649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.802616    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"
Nov 03 21:42:45.811156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.811139    5244 scope.go:122] "RemoveContainer" containerID="a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e"
Nov 03 21:42:45.811498 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:45.811478    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e\": container with ID starting with a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e not found: ID does not exist" containerID="a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e"
Nov 03 21:42:45.811597 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.811509    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e"} err="failed to get container status \"a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e\": rpc error: code = NotFound desc = could not find container \"a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e\": container with ID starting with a8a781baea4306008517a3f7860ae212ae036f2b7f1d84543466390ac2f2d31e not found: ID does not exist"
Nov 03 21:42:45.811597 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.811528    5244 scope.go:122] "RemoveContainer" containerID="3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e"
Nov 03 21:42:45.822437 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.822250    5244 scope.go:122] "RemoveContainer" containerID="3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e"
Nov 03 21:42:45.822731 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:45.822639    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e\": container with ID starting with 3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e not found: ID does not exist" containerID="3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e"
Nov 03 21:42:45.822731 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.822669    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e"} err="failed to get container status \"3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e\": rpc error: code = NotFound desc = could not find container \"3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e\": container with ID starting with 3f12169aca3470cca22b1f2862d10cdefc900413319bb2c72fcfa15d8a90888e not found: ID does not exist"
Nov 03 21:42:45.963723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:45.963683    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"]
Nov 03 21:42:46.012738 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:46.012707    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5733b4-576b968b4c-qr2qv"]
Nov 03 21:42:46.120912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:46.120830    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"]
Nov 03 21:42:46.124017 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:46.123979    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5733b4-665d9fdc76-lm95v"]
Nov 03 21:42:47.136742 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:47.136710    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ae221b-a078-464f-b21e-1efbb8e2d229" path="/var/lib/kubelet/pods/01ae221b-a078-464f-b21e-1efbb8e2d229/volumes"
Nov 03 21:42:47.137144 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:47.137129    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485860b3-cec8-4522-b4f8-8058e8c1e6b0" path="/var/lib/kubelet/pods/485860b3-cec8-4522-b4f8-8058e8c1e6b0/volumes"
Nov 03 21:42:49.383288 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.383237    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"]
Nov 03 21:42:49.384096 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384071    5244 cpu_manager.go:414] "containerMap: removing container" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy"
Nov 03 21:42:49.384201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384098    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy"
Nov 03 21:42:49.384201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384131    5244 cpu_manager.go:414] "containerMap: removing container" podUID="01ae221b-a078-464f-b21e-1efbb8e2d229" containerName="payload-pre-processing"
Nov 03 21:42:49.384201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384140    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ae221b-a078-464f-b21e-1efbb8e2d229" containerName="payload-pre-processing"
Nov 03 21:42:49.384201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384166    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy"
Nov 03 21:42:49.384201 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384174    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384212    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ea56e23a-3305-47a8-b1a1-51c222c2680e" containerName="payload-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384221    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea56e23a-3305-47a8-b1a1-51c222c2680e" containerName="payload-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384235    5244 cpu_manager.go:414] "containerMap: removing container" podUID="485860b3-cec8-4522-b4f8-8058e8c1e6b0" containerName="payload-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384242    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485860b3-cec8-4522-b4f8-8058e8c1e6b0" containerName="payload-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384385    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea56e23a-3305-47a8-b1a1-51c222c2680e" containerName="payload-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384400    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="485860b3-cec8-4522-b4f8-8058e8c1e6b0" containerName="payload-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384410    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="01945076-86c3-42f5-bf87-8cfbe2950a36" containerName="istio-proxy"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384421    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ae221b-a078-464f-b21e-1efbb8e2d229" containerName="payload-pre-processing"
Nov 03 21:42:49.384494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.384433    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad382be0-f946-4823-855a-1bed3a5e9c84" containerName="istio-proxy"
Nov 03 21:42:49.388515 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.388494    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.503546 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.503514    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"]
Nov 03 21:42:49.548100 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548069    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548120    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548229    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548261    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6r9h\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548284    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548347    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548383    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548483    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.548622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.548528    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.649953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.649864    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.649953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.649914    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.649953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.649940    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.649968    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6r9h\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.649989    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650015    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650041    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650089    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650142    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650386    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650429    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650508    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.650720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.650635    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.651067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.651043    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.652538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.652518    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.652673 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.652651    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.709122 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.709086    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:49.709282 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:49.709260    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6r9h\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h\") pod \"e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") " pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:50.002982 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.002950    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:50.436977 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.436951    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"]
Nov 03 21:42:50.438459 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:42:50.438432    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40d7168_bda0_4eea_a436_c7d5df742695.slice/crio-3b2da0484fa2ce0039ea59fc800629edfd4cb8722e16234fee9a9e2f05457d9f WatchSource:0}: Error finding container 3b2da0484fa2ce0039ea59fc800629edfd4cb8722e16234fee9a9e2f05457d9f: Status 404 returned error can't find the container with id 3b2da0484fa2ce0039ea59fc800629edfd4cb8722e16234fee9a9e2f05457d9f
Nov 03 21:42:50.441176 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.441141    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:42:50.441227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.441214    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:42:50.441279 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.441265    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:42:50.966477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.966437    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" event={"ID":"c40d7168-bda0-4eea-a436-c7d5df742695","Type":"ContainerStarted","Data":"f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381"}
Nov 03 21:42:50.966672 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:50.966499    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" event={"ID":"c40d7168-bda0-4eea-a436-c7d5df742695","Type":"ContainerStarted","Data":"3b2da0484fa2ce0039ea59fc800629edfd4cb8722e16234fee9a9e2f05457d9f"}
Nov 03 21:42:51.003417 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:51.003389    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:51.005265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:51.005228    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 21:42:51.005394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:51.005280    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 21:42:51.178363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:51.173949    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podStartSLOduration=2.17392518 podStartE2EDuration="2.17392518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:42:51.170485954 +0000 UTC m=+1642.154482524" watchObservedRunningTime="2026-11-03 21:42:51.17392518 +0000 UTC m=+1642.157921740"
Nov 03 21:42:52.003403 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:52.003365    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 21:42:52.003802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:52.003422    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 21:42:52.466431 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:42:52.466390    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:42:53.003987 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:53.003952    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 21:42:53.004438 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:53.004009    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 21:42:54.004109 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:54.004076    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 21:42:54.004670 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:54.004139    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 21:42:55.004103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:55.004073    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 21:42:55.004278 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:55.004127    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 21:42:56.004148 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:56.004115    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 21:42:56.004391 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:56.004165    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 21:42:56.862143 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:56.862098    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"]
Nov 03 21:42:57.159146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:57.159062    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:57.160384 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:57.160322    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:42:57.160989 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:42:57.160972    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:43:01.863177 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:01.863123    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk" podUID="610eb94f-50c1-4d1f-8a39-67b526733950" containerName="payload-processing" containerID="cri-o://5b7e04d9f046e52c06f2960c31f231f2fee8b20911a46d063fd885c90d75949a" gracePeriod=25
Nov 03 21:43:02.049394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.049352    5244 generic.go:356] "Generic (PLEG): container finished" podID="610eb94f-50c1-4d1f-8a39-67b526733950" containerID="5b7e04d9f046e52c06f2960c31f231f2fee8b20911a46d063fd885c90d75949a" exitCode=0
Nov 03 21:43:02.049541 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.049440    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk" event={"ID":"610eb94f-50c1-4d1f-8a39-67b526733950","Type":"ContainerDied","Data":"5b7e04d9f046e52c06f2960c31f231f2fee8b20911a46d063fd885c90d75949a"}
Nov 03 21:43:02.130314 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.130285    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:43:02.208658 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.208627    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume\") pod \"610eb94f-50c1-4d1f-8a39-67b526733950\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") "
Nov 03 21:43:02.208821 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.208699    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2\" (UniqueName: \"kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2\") pod \"610eb94f-50c1-4d1f-8a39-67b526733950\" (UID: \"610eb94f-50c1-4d1f-8a39-67b526733950\") "
Nov 03 21:43:02.209026 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.209004    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume" pod "610eb94f-50c1-4d1f-8a39-67b526733950" (UID: "610eb94f-50c1-4d1f-8a39-67b526733950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:43:02.209184 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.209170    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/610eb94f-50c1-4d1f-8a39-67b526733950-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:02.211050 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.211023    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2" pod "610eb94f-50c1-4d1f-8a39-67b526733950" (UID: "610eb94f-50c1-4d1f-8a39-67b526733950"). InnerVolumeSpecName "kube-api-access-rrvv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:02.310573 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:02.310531    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvv2\" (UniqueName: \"kubernetes.io/projected/610eb94f-50c1-4d1f-8a39-67b526733950-kube-api-access-rrvv2\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:03.056452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:03.056420    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"
Nov 03 21:43:03.057026 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:03.056454    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk" event={"ID":"610eb94f-50c1-4d1f-8a39-67b526733950","Type":"ContainerDied","Data":"e38df239eb8e33a2e75c0db626757a7c8db90887460f14a5eb8a651763202b23"}
Nov 03 21:43:03.057026 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:03.056528    5244 scope.go:122] "RemoveContainer" containerID="5b7e04d9f046e52c06f2960c31f231f2fee8b20911a46d063fd885c90d75949a"
Nov 03 21:43:03.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:03.219115    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"]
Nov 03 21:43:03.268783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:03.268748    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dad161-6ffc9cfb5d-djprk"]
Nov 03 21:43:05.136972 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:05.136941    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610eb94f-50c1-4d1f-8a39-67b526733950" path="/var/lib/kubelet/pods/610eb94f-50c1-4d1f-8a39-67b526733950/volumes"
Nov 03 21:43:10.816403 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:10.816368    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:43:22.554307 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:43:22.554259    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:43:43.244345 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:43.243747    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"]
Nov 03 21:43:48.245792 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.245725    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld" podUID="0bec8098-2187-4792-abba-d7cd6f8d3f6e" containerName="payload-pre-processing" containerID="cri-o://69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd" gracePeriod=25
Nov 03 21:43:48.750604 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.750565    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:43:48.757046 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.757019    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume\") pod \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") "
Nov 03 21:43:48.757203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.757086    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk\" (UniqueName: \"kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk\") pod \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\" (UID: \"0bec8098-2187-4792-abba-d7cd6f8d3f6e\") "
Nov 03 21:43:48.757395 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.757373    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume" pod "0bec8098-2187-4792-abba-d7cd6f8d3f6e" (UID: "0bec8098-2187-4792-abba-d7cd6f8d3f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:43:48.757584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.757559    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bec8098-2187-4792-abba-d7cd6f8d3f6e-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:48.759275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.759252    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk" pod "0bec8098-2187-4792-abba-d7cd6f8d3f6e" (UID: "0bec8098-2187-4792-abba-d7cd6f8d3f6e"). InnerVolumeSpecName "kube-api-access-jffkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:43:48.858728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:48.858674    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jffkk\" (UniqueName: \"kubernetes.io/projected/0bec8098-2187-4792-abba-d7cd6f8d3f6e-kube-api-access-jffkk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:43:49.389263 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.389216    5244 generic.go:356] "Generic (PLEG): container finished" podID="0bec8098-2187-4792-abba-d7cd6f8d3f6e" containerID="69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd" exitCode=0
Nov 03 21:43:49.389790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.389301    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"
Nov 03 21:43:49.389790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.389299    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld" event={"ID":"0bec8098-2187-4792-abba-d7cd6f8d3f6e","Type":"ContainerDied","Data":"69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd"}
Nov 03 21:43:49.389790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.389428    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld" event={"ID":"0bec8098-2187-4792-abba-d7cd6f8d3f6e","Type":"ContainerDied","Data":"77fdc6e0ca8831eefaa9dbb3897d87aa45381da913a8e05538896a8479949ac7"}
Nov 03 21:43:49.389790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.389459    5244 scope.go:122] "RemoveContainer" containerID="69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd"
Nov 03 21:43:49.402782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.402762    5244 scope.go:122] "RemoveContainer" containerID="69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd"
Nov 03 21:43:49.403154 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:43:49.403118    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd\": container with ID starting with 69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd not found: ID does not exist" containerID="69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd"
Nov 03 21:43:49.403261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.403159    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd"} err="failed to get container status \"69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd\": rpc error: code = NotFound desc = could not find container \"69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd\": container with ID starting with 69118b594ccb5b655466d071460914b183b628993e27d8f77d1500660c7a7dfd not found: ID does not exist"
Nov 03 21:43:49.449759 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.449726    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:43:49.552665 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.552569    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"]
Nov 03 21:43:49.554858 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:49.554836    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fff189-7c9cc55fc5-p7tld"]
Nov 03 21:43:51.136734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:51.136658    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bec8098-2187-4792-abba-d7cd6f8d3f6e" path="/var/lib/kubelet/pods/0bec8098-2187-4792-abba-d7cd6f8d3f6e/volumes"
Nov 03 21:43:52.445082 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:43:52.445042    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:43:57.148643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:57.148604    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"]
Nov 03 21:43:57.149271 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:43:57.148893    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy" containerID="cri-o://7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779" gracePeriod=30
Nov 03 21:44:02.418369 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.418278    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:44:02.488624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.488587    5244 generic.go:356] "Generic (PLEG): container finished" podID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerID="7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779" exitCode=0
Nov 03 21:44:02.488796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.488658    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"
Nov 03 21:44:02.488796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.488679    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" event={"ID":"ae5e5831-49d4-4fc9-adf7-73177d082a13","Type":"ContainerDied","Data":"7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779"}
Nov 03 21:44:02.488796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.488739    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr" event={"ID":"ae5e5831-49d4-4fc9-adf7-73177d082a13","Type":"ContainerDied","Data":"bc43f320cf2f430dfb66496762cb828a9a7c202e228ad21642f0ef4adf4a47b1"}
Nov 03 21:44:02.488796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.488766    5244 scope.go:122] "RemoveContainer" containerID="7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779"
Nov 03 21:44:02.498473 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.498452    5244 scope.go:122] "RemoveContainer" containerID="2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"
Nov 03 21:44:02.507881 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.507859    5244 scope.go:122] "RemoveContainer" containerID="7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779"
Nov 03 21:44:02.508190 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:02.508166    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779\": container with ID starting with 7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779 not found: ID does not exist" containerID="7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779"
Nov 03 21:44:02.508238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.508208    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779"} err="failed to get container status \"7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779\": rpc error: code = NotFound desc = could not find container \"7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779\": container with ID starting with 7ab783dc414fb9016b0390d55c7b1f97d9801e49ca5d44622325251f339c0779 not found: ID does not exist"
Nov 03 21:44:02.508238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.508227    5244 scope.go:122] "RemoveContainer" containerID="2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"
Nov 03 21:44:02.508524 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:02.508504    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b\": container with ID starting with 2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b not found: ID does not exist" containerID="2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"
Nov 03 21:44:02.508578 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.508533    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b"} err="failed to get container status \"2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b\": rpc error: code = NotFound desc = could not find container \"2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b\": container with ID starting with 2a795c66a91967b46bc65355624a2e9c549037f50185830db85225dc2ef9da7b not found: ID does not exist"
Nov 03 21:44:02.567996 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.567951    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.567996 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.567994    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568208 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568125    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568201    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568303 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568279    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568303 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568288    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:02.568492 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568472    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568595 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568519    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:02.568595 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568538    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:02.568595 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568580    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568600    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.568754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568620    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data\") pod \"ae5e5831-49d4-4fc9-adf7-73177d082a13\" (UID: \"ae5e5831-49d4-4fc9-adf7-73177d082a13\") "
Nov 03 21:44:02.569052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.568992    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:02.569119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.569083    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:02.569543 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.569515    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5e5831-49d4-4fc9-adf7-73177d082a13-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.569664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.569547    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.569664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.569565    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.569664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.569582    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.569664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.569596    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.570745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.570723    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:02.570745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.570728    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:02.570882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.570782    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:02.570882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.570835    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd" pod "ae5e5831-49d4-4fc9-adf7-73177d082a13" (UID: "ae5e5831-49d4-4fc9-adf7-73177d082a13"). InnerVolumeSpecName "kube-api-access-pk7dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:02.671130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.671048    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.671130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.671074    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.671130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.671087    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk7dd\" (UniqueName: \"kubernetes.io/projected/ae5e5831-49d4-4fc9-adf7-73177d082a13-kube-api-access-pk7dd\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.671130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.671096    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5e5831-49d4-4fc9-adf7-73177d082a13-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:02.908679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.908641    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"]
Nov 03 21:44:02.913540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:02.913509    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-30f143-openshift-default-79884b6775-fjbnr"]
Nov 03 21:44:03.136389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:03.136360    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" path="/var/lib/kubelet/pods/ae5e5831-49d4-4fc9-adf7-73177d082a13/volumes"
Nov 03 21:44:12.595369 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.595265    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"]
Nov 03 21:44:12.595778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.595519    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy" containerID="cri-o://b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673" gracePeriod=30
Nov 03 21:44:12.599217 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.599185    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"]
Nov 03 21:44:12.599516 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.599470    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy" containerID="cri-o://f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381" gracePeriod=30
Nov 03 21:44:12.623202 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.623171    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"]
Nov 03 21:44:12.623430 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.623393    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" containerID="cri-o://c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6" gracePeriod=30
Nov 03 21:44:12.817187 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.817130    5244 patch_prober.go:29] interesting pod/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:44:12.817538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:12.817499    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:44:14.557184 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:14.557146    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"]
Nov 03 21:44:14.557757 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:14.557438    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6" podUID="16e69ac2-9171-4c52-8b73-54765d2f495e" containerName="maas-api" containerID="cri-o://4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf" gracePeriod=30
Nov 03 21:44:14.999771 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:14.999451    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:44:15.057489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.057453    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\") pod \"16e69ac2-9171-4c52-8b73-54765d2f495e\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") "
Nov 03 21:44:15.057696 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.057518    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca\") pod \"16e69ac2-9171-4c52-8b73-54765d2f495e\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") "
Nov 03 21:44:15.057744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.057722    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps\" (UniqueName: \"kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps\") pod \"16e69ac2-9171-4c52-8b73-54765d2f495e\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") "
Nov 03 21:44:15.057780 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.057760    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs\") pod \"16e69ac2-9171-4c52-8b73-54765d2f495e\" (UID: \"16e69ac2-9171-4c52-8b73-54765d2f495e\") "
Nov 03 21:44:15.058240 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.058205    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca" pod "16e69ac2-9171-4c52-8b73-54765d2f495e" (UID: "16e69ac2-9171-4c52-8b73-54765d2f495e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:15.058384 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.058367    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e69ac2-9171-4c52-8b73-54765d2f495e-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:15.061483 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.061452    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps" pod "16e69ac2-9171-4c52-8b73-54765d2f495e" (UID: "16e69ac2-9171-4c52-8b73-54765d2f495e"). InnerVolumeSpecName "kube-api-access-w78ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:15.081681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.080166    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs" pod "16e69ac2-9171-4c52-8b73-54765d2f495e" (UID: "16e69ac2-9171-4c52-8b73-54765d2f495e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:15.091382 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.088476    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls" pod "16e69ac2-9171-4c52-8b73-54765d2f495e" (UID: "16e69ac2-9171-4c52-8b73-54765d2f495e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:15.159900 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.159854    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:15.159900 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.159897    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w78ps\" (UniqueName: \"kubernetes.io/projected/16e69ac2-9171-4c52-8b73-54765d2f495e-kube-api-access-w78ps\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:15.160168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.159914    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e69ac2-9171-4c52-8b73-54765d2f495e-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:15.584620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.583210    5244 generic.go:356] "Generic (PLEG): container finished" podID="16e69ac2-9171-4c52-8b73-54765d2f495e" containerID="4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf" exitCode=0
Nov 03 21:44:15.584620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.583423    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6" event={"ID":"16e69ac2-9171-4c52-8b73-54765d2f495e","Type":"ContainerDied","Data":"4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf"}
Nov 03 21:44:15.584620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.583487    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6" event={"ID":"16e69ac2-9171-4c52-8b73-54765d2f495e","Type":"ContainerDied","Data":"c6db4657f973b2bb5727095df2298cf3dd31eefc4f35a27085c1c2fc0199b9da"}
Nov 03 21:44:15.584620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.583534    5244 scope.go:122] "RemoveContainer" containerID="4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf"
Nov 03 21:44:15.584620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.583750    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"
Nov 03 21:44:15.646662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.646632    5244 scope.go:122] "RemoveContainer" containerID="4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf"
Nov 03 21:44:15.647036 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:15.647007    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf\": container with ID starting with 4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf not found: ID does not exist" containerID="4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf"
Nov 03 21:44:15.647150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.647042    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf"} err="failed to get container status \"4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf\": rpc error: code = NotFound desc = could not find container \"4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf\": container with ID starting with 4f7e26c5e05487acbc8b78af760c3a92b202f07beb7f9a8b8328c50c90a7ccdf not found: ID does not exist"
Nov 03 21:44:15.749317 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.749277    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"]
Nov 03 21:44:15.798954 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:15.798914    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5733b4-84f74459-22kj6"]
Nov 03 21:44:17.142810 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:17.142774    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e69ac2-9171-4c52-8b73-54765d2f495e" path="/var/lib/kubelet/pods/16e69ac2-9171-4c52-8b73-54765d2f495e/volumes"
Nov 03 21:44:17.901486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:17.901459    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:44:17.930882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:17.930856    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:44:17.967394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:17.967367    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:44:18.013057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013019    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013057    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013082    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013107    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013128    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013171    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013203    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013237    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013292    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013343    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013386    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013450    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013453    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013522    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013543    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013591    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013617    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013672    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013675    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.013702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013677    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013758    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013793    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs\") pod \"c5557439-52d9-4821-b6e3-e8df9649a051\" (UID: \"c5557439-52d9-4821-b6e3-e8df9649a051\") "
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013861    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013875    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo\") pod \"c40d7168-bda0-4eea-a436-c7d5df742695\" (UID: \"c40d7168-bda0-4eea-a436-c7d5df742695\") "
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.013884    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014060    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014059    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.014255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014093    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014670    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014694    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014712    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014729    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014744    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5557439-52d9-4821-b6e3-e8df9649a051-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014759    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014775    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d7168-bda0-4eea-a436-c7d5df742695-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014791    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014805    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.014911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.014821    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.016545 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.016503    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.016892 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.016864    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.016987 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.016891    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "kube-api-access-t6r9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.016987 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.016901    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.016987 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.016981    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.017112 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.017006    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo" pod "c40d7168-bda0-4eea-a436-c7d5df742695" (UID: "c40d7168-bda0-4eea-a436-c7d5df742695"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.017112 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.017015    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "kube-api-access-6lrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.017689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.017674    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token" pod "c5557439-52d9-4821-b6e3-e8df9649a051" (UID: "c5557439-52d9-4821-b6e3-e8df9649a051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.116034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.115992    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116216 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116116    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116216 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116173    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116216 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116207    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116257    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116291    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116407 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116317    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116448    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.116571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116496    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data\") pod \"acc93a84-e1d3-479f-bee2-26843f2214f6\" (UID: \"acc93a84-e1d3-479f-bee2-26843f2214f6\") "
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.116918    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117009    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5557439-52d9-4821-b6e3-e8df9649a051-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117034    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lrqs\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-kube-api-access-6lrqs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117051    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d7168-bda0-4eea-a436-c7d5df742695-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117065    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5557439-52d9-4821-b6e3-e8df9649a051-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117078    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d7168-bda0-4eea-a436-c7d5df742695-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117091    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6r9h\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-kube-api-access-t6r9h\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117104    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d7168-bda0-4eea-a436-c7d5df742695-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117141    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5557439-52d9-4821-b6e3-e8df9649a051-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.117266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117204    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.117875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117291    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.117875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117363    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.117875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.117520    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.119291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.119260    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:18.119432 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.119412    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.119556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.119524    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:18.120071 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.120028    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6" pod "acc93a84-e1d3-479f-bee2-26843f2214f6" (UID: "acc93a84-e1d3-479f-bee2-26843f2214f6"). InnerVolumeSpecName "kube-api-access-9l7w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:18.218742 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218696    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218788    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l7w6\" (UniqueName: \"kubernetes.io/projected/acc93a84-e1d3-479f-bee2-26843f2214f6-kube-api-access-9l7w6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218811    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218824    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218835    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218845    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218854    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc93a84-e1d3-479f-bee2-26843f2214f6-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218869    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc93a84-e1d3-479f-bee2-26843f2214f6-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.219150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.218881    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc93a84-e1d3-479f-bee2-26843f2214f6-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:18.615704 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.615613    5244 generic.go:356] "Generic (PLEG): container finished" podID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerID="c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6" exitCode=0
Nov 03 21:44:18.615704 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.615676    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"
Nov 03 21:44:18.615934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.615707    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" event={"ID":"acc93a84-e1d3-479f-bee2-26843f2214f6","Type":"ContainerDied","Data":"c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6"}
Nov 03 21:44:18.615934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.615780    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd" event={"ID":"acc93a84-e1d3-479f-bee2-26843f2214f6","Type":"ContainerDied","Data":"75536710aec30f0828cdfdb7a1a367d3ef599841d00231a54c8c3f54e6911f7e"}
Nov 03 21:44:18.615934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.615816    5244 scope.go:122] "RemoveContainer" containerID="c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6"
Nov 03 21:44:18.617388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.617359    5244 generic.go:356] "Generic (PLEG): container finished" podID="c5557439-52d9-4821-b6e3-e8df9649a051" containerID="b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673" exitCode=0
Nov 03 21:44:18.617511 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.617416    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"
Nov 03 21:44:18.617511 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.617483    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" event={"ID":"c5557439-52d9-4821-b6e3-e8df9649a051","Type":"ContainerDied","Data":"b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673"}
Nov 03 21:44:18.617674 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.617530    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd" event={"ID":"c5557439-52d9-4821-b6e3-e8df9649a051","Type":"ContainerDied","Data":"ccf50d5dbc3a392ca7b877fefa2cc4094b8c88b15970ee3324cf998bbd8677b5"}
Nov 03 21:44:18.619252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.619224    5244 generic.go:356] "Generic (PLEG): container finished" podID="c40d7168-bda0-4eea-a436-c7d5df742695" containerID="f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381" exitCode=0
Nov 03 21:44:18.619372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.619260    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" event={"ID":"c40d7168-bda0-4eea-a436-c7d5df742695","Type":"ContainerDied","Data":"f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381"}
Nov 03 21:44:18.619372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.619295    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"
Nov 03 21:44:18.619372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.619318    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp" event={"ID":"c40d7168-bda0-4eea-a436-c7d5df742695","Type":"ContainerDied","Data":"3b2da0484fa2ce0039ea59fc800629edfd4cb8722e16234fee9a9e2f05457d9f"}
Nov 03 21:44:18.645447 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.645429    5244 scope.go:122] "RemoveContainer" containerID="c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6"
Nov 03 21:44:18.645808 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:18.645787    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6\": container with ID starting with c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6 not found: ID does not exist" containerID="c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6"
Nov 03 21:44:18.645875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.645821    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6"} err="failed to get container status \"c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6\": rpc error: code = NotFound desc = could not find container \"c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6\": container with ID starting with c06934f550eb9ceef9a9095ddb2a708477a8091408f7230f6c82bdec95e4dcd6 not found: ID does not exist"
Nov 03 21:44:18.645875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.645841    5244 scope.go:122] "RemoveContainer" containerID="b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673"
Nov 03 21:44:18.655520 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.655498    5244 scope.go:122] "RemoveContainer" containerID="b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673"
Nov 03 21:44:18.655807 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:18.655790    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673\": container with ID starting with b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673 not found: ID does not exist" containerID="b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673"
Nov 03 21:44:18.655848 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.655817    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673"} err="failed to get container status \"b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673\": rpc error: code = NotFound desc = could not find container \"b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673\": container with ID starting with b5bf986712d3266dcec0b8b86209de34ce64089da21fe588bcc2e1e05c821673 not found: ID does not exist"
Nov 03 21:44:18.655848 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.655837    5244 scope.go:122] "RemoveContainer" containerID="f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381"
Nov 03 21:44:18.664828 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.664806    5244 scope.go:122] "RemoveContainer" containerID="f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381"
Nov 03 21:44:18.665111 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:18.665094    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381\": container with ID starting with f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381 not found: ID does not exist" containerID="f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381"
Nov 03 21:44:18.665160 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.665118    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381"} err="failed to get container status \"f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381\": rpc error: code = NotFound desc = could not find container \"f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381\": container with ID starting with f925416744d0b9e3ca91e8bc1ec57fd21266021c4e7ef75c310f2b3b85d66381 not found: ID does not exist"
Nov 03 21:44:18.778870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.778834    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"]
Nov 03 21:44:18.827750 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.827709    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-4x6hd"]
Nov 03 21:44:18.890029 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.889950    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"]
Nov 03 21:44:18.893619 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:18.893590    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-w7mcp"]
Nov 03 21:44:19.002239 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:19.002205    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"]
Nov 03 21:44:19.009081 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:19.009053    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1662df5-openshift-default-6dc7765f97-qfdcd"]
Nov 03 21:44:19.150445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:19.150303    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" path="/var/lib/kubelet/pods/acc93a84-e1d3-479f-bee2-26843f2214f6/volumes"
Nov 03 21:44:19.150898 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:19.150877    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" path="/var/lib/kubelet/pods/c40d7168-bda0-4eea-a436-c7d5df742695/volumes"
Nov 03 21:44:19.151522 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:19.151499    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" path="/var/lib/kubelet/pods/c5557439-52d9-4821-b6e3-e8df9649a051/volumes"
Nov 03 21:44:22.502353 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:22.498532    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:44:23.895624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:23.895589    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"]
Nov 03 21:44:23.896042 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:23.895828    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy" containerID="cri-o://94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd" gracePeriod=30
Nov 03 21:44:29.155795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.155769    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:44:29.179689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179612    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.179689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179663    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.179860 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179726    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.179860 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179758    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.179860 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179851    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.179976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179941    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.180029 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.179987    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.180029 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180017    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.180130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180021    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:29.180130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180069    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket\") pod \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\" (UID: \"b1e638c9-bb2f-4356-ba2b-3110b7000a9d\") "
Nov 03 21:44:29.180130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180067    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:29.180305 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180283    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:29.180436 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180203    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:29.180526 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180480    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:29.180773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180752    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.180842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180781    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.180842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180798    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.180842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180813    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.180842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.180828    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.182104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.182080    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "kube-api-access-4sj7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:29.182236 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.182106    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:44:29.182236 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.182193    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:44:29.182415 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.182399    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token" pod "b1e638c9-bb2f-4356-ba2b-3110b7000a9d" (UID: "b1e638c9-bb2f-4356-ba2b-3110b7000a9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:29.281964 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.281930    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.281964 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.281960    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.281964 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.281971    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.282203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.281981    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sj7x\" (UniqueName: \"kubernetes.io/projected/b1e638c9-bb2f-4356-ba2b-3110b7000a9d-kube-api-access-4sj7x\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:29.699250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.699200    5244 generic.go:356] "Generic (PLEG): container finished" podID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerID="94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd" exitCode=0
Nov 03 21:44:29.699452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.699266    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"
Nov 03 21:44:29.699452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.699282    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" event={"ID":"b1e638c9-bb2f-4356-ba2b-3110b7000a9d","Type":"ContainerDied","Data":"94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd"}
Nov 03 21:44:29.699452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.699400    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw" event={"ID":"b1e638c9-bb2f-4356-ba2b-3110b7000a9d","Type":"ContainerDied","Data":"48382f15f0f517e2aa358c2194dc384562bf0da809a331a49d8232851ff1b7be"}
Nov 03 21:44:29.699452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.699432    5244 scope.go:122] "RemoveContainer" containerID="94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd"
Nov 03 21:44:29.709033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.709013    5244 scope.go:122] "RemoveContainer" containerID="fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"
Nov 03 21:44:29.717762 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.717746    5244 scope.go:122] "RemoveContainer" containerID="94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd"
Nov 03 21:44:29.718071 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:29.718050    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd\": container with ID starting with 94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd not found: ID does not exist" containerID="94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd"
Nov 03 21:44:29.718109 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.718082    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd"} err="failed to get container status \"94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd\": rpc error: code = NotFound desc = could not find container \"94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd\": container with ID starting with 94540860befcfa81697985d42827390b93b567eaba4216ccf94dac93de9948fd not found: ID does not exist"
Nov 03 21:44:29.718109 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.718101    5244 scope.go:122] "RemoveContainer" containerID="fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"
Nov 03 21:44:29.718485 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:29.718466    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a\": container with ID starting with fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a not found: ID does not exist" containerID="fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"
Nov 03 21:44:29.718521 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.718494    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a"} err="failed to get container status \"fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a\": rpc error: code = NotFound desc = could not find container \"fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a\": container with ID starting with fcd5f4c3b6b38dc120beae45f19cd80538d217dab04cfbe0f4eca917ad2aa81a not found: ID does not exist"
Nov 03 21:44:29.862728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.862647    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"]
Nov 03 21:44:29.865946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:29.865910    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5733b4-openshift-default-54dd7949d7-q4kzw"]
Nov 03 21:44:31.135736 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:31.135704    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" path="/var/lib/kubelet/pods/b1e638c9-bb2f-4356-ba2b-3110b7000a9d/volumes"
Nov 03 21:44:34.287255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:34.287216    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:44:36.911212 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:36.911179    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"]
Nov 03 21:44:36.911711 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:36.911460    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8" podUID="d2377367-0055-4a96-84b6-b96f77a3ea32" containerName="maas-api" containerID="cri-o://0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103" gracePeriod=30
Nov 03 21:44:37.236371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.233951    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:44:37.302035 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.301993    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb\" (UniqueName: \"kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb\") pod \"d2377367-0055-4a96-84b6-b96f77a3ea32\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") "
Nov 03 21:44:37.302254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.302048    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls\") pod \"d2377367-0055-4a96-84b6-b96f77a3ea32\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") "
Nov 03 21:44:37.302254 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.302247    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs\") pod \"d2377367-0055-4a96-84b6-b96f77a3ea32\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") "
Nov 03 21:44:37.302397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.302301    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca\") pod \"d2377367-0055-4a96-84b6-b96f77a3ea32\" (UID: \"d2377367-0055-4a96-84b6-b96f77a3ea32\") "
Nov 03 21:44:37.302680 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.302651    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca" pod "d2377367-0055-4a96-84b6-b96f77a3ea32" (UID: "d2377367-0055-4a96-84b6-b96f77a3ea32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:44:37.302867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.302840    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2377367-0055-4a96-84b6-b96f77a3ea32-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:37.304610 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.304583    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs" pod "d2377367-0055-4a96-84b6-b96f77a3ea32" (UID: "d2377367-0055-4a96-84b6-b96f77a3ea32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:37.304710 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.304635    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls" pod "d2377367-0055-4a96-84b6-b96f77a3ea32" (UID: "d2377367-0055-4a96-84b6-b96f77a3ea32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:44:37.305019 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.304997    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb" pod "d2377367-0055-4a96-84b6-b96f77a3ea32" (UID: "d2377367-0055-4a96-84b6-b96f77a3ea32"). InnerVolumeSpecName "kube-api-access-497tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:44:37.436793 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.433308    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:37.436793 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.433357    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497tb\" (UniqueName: \"kubernetes.io/projected/d2377367-0055-4a96-84b6-b96f77a3ea32-kube-api-access-497tb\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:37.436793 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.433370    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2377367-0055-4a96-84b6-b96f77a3ea32-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:44:37.748795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.748753    5244 generic.go:356] "Generic (PLEG): container finished" podID="d2377367-0055-4a96-84b6-b96f77a3ea32" containerID="0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103" exitCode=0
Nov 03 21:44:37.749015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.748802    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"
Nov 03 21:44:37.749015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.748804    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8" event={"ID":"d2377367-0055-4a96-84b6-b96f77a3ea32","Type":"ContainerDied","Data":"0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103"}
Nov 03 21:44:37.749015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.748866    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8" event={"ID":"d2377367-0055-4a96-84b6-b96f77a3ea32","Type":"ContainerDied","Data":"5dde3667c8961f8f40b6c7191fcaffffd6429810d6fdd9d5eee1d54d28fb1621"}
Nov 03 21:44:37.749015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.748897    5244 scope.go:122] "RemoveContainer" containerID="0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103"
Nov 03 21:44:37.760080 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.760054    5244 scope.go:122] "RemoveContainer" containerID="0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103"
Nov 03 21:44:37.760424 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:37.760405    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103\": container with ID starting with 0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103 not found: ID does not exist" containerID="0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103"
Nov 03 21:44:37.760518 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.760434    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103"} err="failed to get container status \"0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103\": rpc error: code = NotFound desc = could not find container \"0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103\": container with ID starting with 0393dd8d61cee72cb91d2fdcce697d1f3f94ef7a42c9cf6bcb9b82f11514a103 not found: ID does not exist"
Nov 03 21:44:37.936355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.918201    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"]
Nov 03 21:44:37.977067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:37.977028    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fff189-564c5496f4-5l4t8"]
Nov 03 21:44:39.137445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:44:39.137409    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2377367-0055-4a96-84b6-b96f77a3ea32" path="/var/lib/kubelet/pods/d2377367-0055-4a96-84b6-b96f77a3ea32/volumes"
Nov 03 21:44:52.438156 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:44:52.438115    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:45:00.173353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.159053    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"]
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193884    5244 cpu_manager.go:414] "containerMap: removing container" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193914    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193938    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193946    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193967    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0bec8098-2187-4792-abba-d7cd6f8d3f6e" containerName="payload-pre-processing"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193974    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bec8098-2187-4792-abba-d7cd6f8d3f6e" containerName="payload-pre-processing"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193987    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.193996    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194021    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194029    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194054    5244 cpu_manager.go:414] "containerMap: removing container" podUID="610eb94f-50c1-4d1f-8a39-67b526733950" containerName="payload-processing"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194062    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610eb94f-50c1-4d1f-8a39-67b526733950" containerName="payload-processing"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194071    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d2377367-0055-4a96-84b6-b96f77a3ea32" containerName="maas-api"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194079    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2377367-0055-4a96-84b6-b96f77a3ea32" containerName="maas-api"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194096    5244 cpu_manager.go:414] "containerMap: removing container" podUID="16e69ac2-9171-4c52-8b73-54765d2f495e" containerName="maas-api"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194103    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e69ac2-9171-4c52-8b73-54765d2f495e" containerName="maas-api"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194112    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194119    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194143    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194151    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194257    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5557439-52d9-4821-b6e3-e8df9649a051" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194273    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="610eb94f-50c1-4d1f-8a39-67b526733950" containerName="payload-processing"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194282    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2377367-0055-4a96-84b6-b96f77a3ea32" containerName="maas-api"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194291    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e69ac2-9171-4c52-8b73-54765d2f495e" containerName="maas-api"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194299    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194310    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194320    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5e5831-49d4-4fc9-adf7-73177d082a13" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194349    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc93a84-e1d3-479f-bee2-26843f2214f6" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194358    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40d7168-bda0-4eea-a436-c7d5df742695" containerName="istio-proxy"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194369    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bec8098-2187-4792-abba-d7cd6f8d3f6e" containerName="payload-pre-processing"
Nov 03 21:45:00.194353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.194377    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy"
Nov 03 21:45:00.201581 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.201552    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"]
Nov 03 21:45:00.202449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.202424    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy"
Nov 03 21:45:00.205949 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.205908    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e638c9-bb2f-4356-ba2b-3110b7000a9d" containerName="istio-proxy"
Nov 03 21:45:00.208196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.203470    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.211481 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.211463    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.216400 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.216370    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdfk\" (UniqueName: \"kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk\") pod \"maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.216498 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.216423    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6v2w\" (UniqueName: \"kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w\") pod \"maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.216607 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.216581    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.216660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.216642    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.283355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.282903    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"]
Nov 03 21:45:00.318624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.318575    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdfk\" (UniqueName: \"kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk\") pod \"maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.318624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.318629    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6v2w\" (UniqueName: \"kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w\") pod \"maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.318906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.318682    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.318906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.318722    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.321065 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.319592    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.321065 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.320750    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.354364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.352747    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:45:00.362357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.355431    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:45:00.362357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.355521    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:45:00.421554 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.421502    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-thkfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:45:00.421749 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.421574    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:45:00.423010 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.422985    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"]
Nov 03 21:45:00.431103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.431004    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6v2w\" (UniqueName: \"kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w\") pod \"maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.440510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.440440    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdfk\" (UniqueName: \"kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk\") pod \"maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:00.530721 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.530507    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:00.584387 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:00.583555    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:01.078972 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:01.078938    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"]
Nov 03 21:45:01.120663 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:45:01.120630    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda656bb03_0a72_497c_9610_4a709fb63eb9.slice/crio-4a4241cf5c916fb1dc0ae405abe50fd96b7694b6edd6512dfd5410bd75be8a31 WatchSource:0}: Error finding container 4a4241cf5c916fb1dc0ae405abe50fd96b7694b6edd6512dfd5410bd75be8a31: Status 404 returned error can't find the container with id 4a4241cf5c916fb1dc0ae405abe50fd96b7694b6edd6512dfd5410bd75be8a31
Nov 03 21:45:01.127573 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:01.127546    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"]
Nov 03 21:45:01.132027 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:45:01.131997    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2974da9b_a2fd_4290_b8fe_99baf4106285.slice/crio-68a28aa5a341fcc89c3b5289b232e3c58df314432949ea85e6117a88b19287fc WatchSource:0}: Error finding container 68a28aa5a341fcc89c3b5289b232e3c58df314432949ea85e6117a88b19287fc: Status 404 returned error can't find the container with id 68a28aa5a341fcc89c3b5289b232e3c58df314432949ea85e6117a88b19287fc
Nov 03 21:45:02.013795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.013751    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" event={"ID":"2974da9b-a2fd-4290-b8fe-99baf4106285","Type":"ContainerStarted","Data":"dcf8f9b52ff1dfc11ed22f139d2fcea74c2b8148d2823f8f8e5f60df18b22bfd"}
Nov 03 21:45:02.014292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.013844    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" event={"ID":"2974da9b-a2fd-4290-b8fe-99baf4106285","Type":"ContainerStarted","Data":"68a28aa5a341fcc89c3b5289b232e3c58df314432949ea85e6117a88b19287fc"}
Nov 03 21:45:02.015572 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.015545    5244 generic.go:356] "Generic (PLEG): container finished" podID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerID="67d01054e37ec2e3621101c4a9cff87182cca62cfc00d5119b8cc5174fa96621" exitCode=7
Nov 03 21:45:02.015694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.015607    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" event={"ID":"a656bb03-0a72-497c-9610-4a709fb63eb9","Type":"ContainerDied","Data":"67d01054e37ec2e3621101c4a9cff87182cca62cfc00d5119b8cc5174fa96621"}
Nov 03 21:45:02.015694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.015638    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" event={"ID":"a656bb03-0a72-497c-9610-4a709fb63eb9","Type":"ContainerStarted","Data":"4a4241cf5c916fb1dc0ae405abe50fd96b7694b6edd6512dfd5410bd75be8a31"}
Nov 03 21:45:02.015835 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.015821    5244 scope.go:122] "RemoveContainer" containerID="67d01054e37ec2e3621101c4a9cff87182cca62cfc00d5119b8cc5174fa96621"
Nov 03 21:45:02.130564 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:02.130512    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" podStartSLOduration=2.130497603 podStartE2EDuration="2.130497603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:45:02.127986183 +0000 UTC m=+1773.111982742" watchObservedRunningTime="2026-11-03 21:45:02.130497603 +0000 UTC m=+1773.114494162"
Nov 03 21:45:03.023184 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.023145    5244 generic.go:356] "Generic (PLEG): container finished" podID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerID="dcf8f9b52ff1dfc11ed22f139d2fcea74c2b8148d2823f8f8e5f60df18b22bfd" exitCode=60
Nov 03 21:45:03.023666 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.023269    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" event={"ID":"2974da9b-a2fd-4290-b8fe-99baf4106285","Type":"ContainerDied","Data":"dcf8f9b52ff1dfc11ed22f139d2fcea74c2b8148d2823f8f8e5f60df18b22bfd"}
Nov 03 21:45:03.023666 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.023553    5244 scope.go:122] "RemoveContainer" containerID="dcf8f9b52ff1dfc11ed22f139d2fcea74c2b8148d2823f8f8e5f60df18b22bfd"
Nov 03 21:45:03.025606 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.025575    5244 generic.go:356] "Generic (PLEG): container finished" podID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerID="f7d8e4027dbb08fec6e01d2e25fe45e75676694a37f0d9679db8995a9714d1c7" exitCode=7
Nov 03 21:45:03.025717 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.025637    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" event={"ID":"a656bb03-0a72-497c-9610-4a709fb63eb9","Type":"ContainerDied","Data":"f7d8e4027dbb08fec6e01d2e25fe45e75676694a37f0d9679db8995a9714d1c7"}
Nov 03 21:45:03.025717 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.025695    5244 scope.go:122] "RemoveContainer" containerID="67d01054e37ec2e3621101c4a9cff87182cca62cfc00d5119b8cc5174fa96621"
Nov 03 21:45:03.025928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:03.025914    5244 scope.go:122] "RemoveContainer" containerID="f7d8e4027dbb08fec6e01d2e25fe45e75676694a37f0d9679db8995a9714d1c7"
Nov 03 21:45:03.026600 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:03.026563    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz_odh-ai-gateway-infra(a656bb03-0a72-497c-9610-4a709fb63eb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9"
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:04.046261    5244 scope.go:122] "RemoveContainer" containerID="f7d8e4027dbb08fec6e01d2e25fe45e75676694a37f0d9679db8995a9714d1c7"
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:04.046671    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz_odh-ai-gateway-infra(a656bb03-0a72-497c-9610-4a709fb63eb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9"
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:04.049602    5244 generic.go:356] "Generic (PLEG): container finished" podID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerID="c863f278e348b63c5564f9980cb394ebc19a839534f937ab9834305c4bebb778" exitCode=60
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:04.049644    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" event={"ID":"2974da9b-a2fd-4290-b8fe-99baf4106285","Type":"ContainerDied","Data":"c863f278e348b63c5564f9980cb394ebc19a839534f937ab9834305c4bebb778"}
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:04.049697    5244 scope.go:122] "RemoveContainer" containerID="dcf8f9b52ff1dfc11ed22f139d2fcea74c2b8148d2823f8f8e5f60df18b22bfd"
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:04.050129    5244 scope.go:122] "RemoveContainer" containerID="c863f278e348b63c5564f9980cb394ebc19a839534f937ab9834305c4bebb778"
Nov 03 21:45:04.054186 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:04.051996    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25_odh-ai-gateway-infra(2974da9b-a2fd-4290-b8fe-99baf4106285)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285"
Nov 03 21:45:05.058083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:05.058056    5244 scope.go:122] "RemoveContainer" containerID="c863f278e348b63c5564f9980cb394ebc19a839534f937ab9834305c4bebb778"
Nov 03 21:45:05.058480 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:05.058320    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25_odh-ai-gateway-infra(2974da9b-a2fd-4290-b8fe-99baf4106285)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285"
Nov 03 21:45:17.129347 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:17.129298    5244 scope.go:122] "RemoveContainer" containerID="f7d8e4027dbb08fec6e01d2e25fe45e75676694a37f0d9679db8995a9714d1c7"
Nov 03 21:45:17.129755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:17.129410    5244 scope.go:122] "RemoveContainer" containerID="c863f278e348b63c5564f9980cb394ebc19a839534f937ab9834305c4bebb778"
Nov 03 21:45:18.129145 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.129105    5244 generic.go:356] "Generic (PLEG): container finished" podID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerID="7e62ccd24a69446f2493233f96ea418f2d41685370a027618c895e0a56a35063" exitCode=60
Nov 03 21:45:18.129412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.129171    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" event={"ID":"2974da9b-a2fd-4290-b8fe-99baf4106285","Type":"ContainerDied","Data":"7e62ccd24a69446f2493233f96ea418f2d41685370a027618c895e0a56a35063"}
Nov 03 21:45:18.129412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.129228    5244 scope.go:122] "RemoveContainer" containerID="c863f278e348b63c5564f9980cb394ebc19a839534f937ab9834305c4bebb778"
Nov 03 21:45:18.129720 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.129633    5244 scope.go:122] "RemoveContainer" containerID="7e62ccd24a69446f2493233f96ea418f2d41685370a027618c895e0a56a35063"
Nov 03 21:45:18.129918 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:18.129901    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25_odh-ai-gateway-infra(2974da9b-a2fd-4290-b8fe-99baf4106285)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285"
Nov 03 21:45:18.131907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.131887    5244 generic.go:356] "Generic (PLEG): container finished" podID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerID="52db33aa8a66ba8f2d0fe7195d93577bdd3afb609f574e420a590d32ceca81b3" exitCode=60
Nov 03 21:45:18.131983 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.131960    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" event={"ID":"a656bb03-0a72-497c-9610-4a709fb63eb9","Type":"ContainerDied","Data":"52db33aa8a66ba8f2d0fe7195d93577bdd3afb609f574e420a590d32ceca81b3"}
Nov 03 21:45:18.132364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.132347    5244 scope.go:122] "RemoveContainer" containerID="52db33aa8a66ba8f2d0fe7195d93577bdd3afb609f574e420a590d32ceca81b3"
Nov 03 21:45:18.132628 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:18.132609    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz_odh-ai-gateway-infra(a656bb03-0a72-497c-9610-4a709fb63eb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9"
Nov 03 21:45:18.153497 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:18.153478    5244 scope.go:122] "RemoveContainer" containerID="f7d8e4027dbb08fec6e01d2e25fe45e75676694a37f0d9679db8995a9714d1c7"
Nov 03 21:45:19.173236 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.173193    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"]
Nov 03 21:45:19.173728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.173422    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" podUID="961af80d-53aa-43f4-b164-d331e017aea4" containerName="maas-api" containerID="cri-o://1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726" gracePeriod=30
Nov 03 21:45:19.273701 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.273666    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"]
Nov 03 21:45:19.367400 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.364908    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"]
Nov 03 21:45:19.715083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.715041    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:19.762990 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.762700    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:45:19.803433 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.803400    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\") pod \"961af80d-53aa-43f4-b164-d331e017aea4\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") "
Nov 03 21:45:19.803602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.803471    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb\" (UniqueName: \"kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb\") pod \"961af80d-53aa-43f4-b164-d331e017aea4\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") "
Nov 03 21:45:19.803602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.803540    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w\" (UniqueName: \"kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w\") pod \"2974da9b-a2fd-4290-b8fe-99baf4106285\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") "
Nov 03 21:45:19.803722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.803633    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca\") pod \"961af80d-53aa-43f4-b164-d331e017aea4\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") "
Nov 03 21:45:19.803722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.803664    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs\") pod \"961af80d-53aa-43f4-b164-d331e017aea4\" (UID: \"961af80d-53aa-43f4-b164-d331e017aea4\") "
Nov 03 21:45:19.803828 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.803784    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca\") pod \"2974da9b-a2fd-4290-b8fe-99baf4106285\" (UID: \"2974da9b-a2fd-4290-b8fe-99baf4106285\") "
Nov 03 21:45:19.804251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.804204    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca" pod "961af80d-53aa-43f4-b164-d331e017aea4" (UID: "961af80d-53aa-43f4-b164-d331e017aea4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.804251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.804202    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca" pod "2974da9b-a2fd-4290-b8fe-99baf4106285" (UID: "2974da9b-a2fd-4290-b8fe-99baf4106285"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.806556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.806525    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls" pod "961af80d-53aa-43f4-b164-d331e017aea4" (UID: "961af80d-53aa-43f4-b164-d331e017aea4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:45:19.806692 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.806557    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w" pod "2974da9b-a2fd-4290-b8fe-99baf4106285" (UID: "2974da9b-a2fd-4290-b8fe-99baf4106285"). InnerVolumeSpecName "kube-api-access-g6v2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.806692 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.806584    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb" pod "961af80d-53aa-43f4-b164-d331e017aea4" (UID: "961af80d-53aa-43f4-b164-d331e017aea4"). InnerVolumeSpecName "kube-api-access-nplsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:19.806885 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.806868    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs" pod "961af80d-53aa-43f4-b164-d331e017aea4" (UID: "961af80d-53aa-43f4-b164-d331e017aea4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:45:19.825842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.825821    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:19.905169 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905087    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk\" (UniqueName: \"kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk\") pod \"a656bb03-0a72-497c-9610-4a709fb63eb9\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") "
Nov 03 21:45:19.905169 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905159    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca\") pod \"a656bb03-0a72-497c-9610-4a709fb63eb9\" (UID: \"a656bb03-0a72-497c-9610-4a709fb63eb9\") "
Nov 03 21:45:19.905574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905541    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.905664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905587    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nplsb\" (UniqueName: \"kubernetes.io/projected/961af80d-53aa-43f4-b164-d331e017aea4-kube-api-access-nplsb\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.905664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905607    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6v2w\" (UniqueName: \"kubernetes.io/projected/2974da9b-a2fd-4290-b8fe-99baf4106285-kube-api-access-g6v2w\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.905664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905623    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961af80d-53aa-43f4-b164-d331e017aea4-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.905664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905638    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961af80d-53aa-43f4-b164-d331e017aea4-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.905664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905653    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974da9b-a2fd-4290-b8fe-99baf4106285-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:19.905664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.905556    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca" pod "a656bb03-0a72-497c-9610-4a709fb63eb9" (UID: "a656bb03-0a72-497c-9610-4a709fb63eb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:45:19.907882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:19.907854    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk" pod "a656bb03-0a72-497c-9610-4a709fb63eb9" (UID: "a656bb03-0a72-497c-9610-4a709fb63eb9"). InnerVolumeSpecName "kube-api-access-wtdfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:45:20.007139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.007102    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdfk\" (UniqueName: \"kubernetes.io/projected/a656bb03-0a72-497c-9610-4a709fb63eb9-kube-api-access-wtdfk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.007139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.007133    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a656bb03-0a72-497c-9610-4a709fb63eb9-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:45:20.218449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.218407    5244 generic.go:356] "Generic (PLEG): container finished" podID="961af80d-53aa-43f4-b164-d331e017aea4" containerID="1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726" exitCode=0
Nov 03 21:45:20.219106 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.219079    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" event={"ID":"961af80d-53aa-43f4-b164-d331e017aea4","Type":"ContainerDied","Data":"1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726"}
Nov 03 21:45:20.219283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.219268    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb" event={"ID":"961af80d-53aa-43f4-b164-d331e017aea4","Type":"ContainerDied","Data":"1c5c4a8a4d35dd1ce8996d9dd9e86031aeab287e18e286ed2c3e0b904493d849"}
Nov 03 21:45:20.219424 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.219410    5244 scope.go:122] "RemoveContainer" containerID="1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726"
Nov 03 21:45:20.219546 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.219523    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"
Nov 03 21:45:20.224830 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.224804    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25" event={"ID":"2974da9b-a2fd-4290-b8fe-99baf4106285","Type":"ContainerDied","Data":"68a28aa5a341fcc89c3b5289b232e3c58df314432949ea85e6117a88b19287fc"}
Nov 03 21:45:20.224954 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.224893    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"
Nov 03 21:45:20.230644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.230131    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz" event={"ID":"a656bb03-0a72-497c-9610-4a709fb63eb9","Type":"ContainerDied","Data":"4a4241cf5c916fb1dc0ae405abe50fd96b7694b6edd6512dfd5410bd75be8a31"}
Nov 03 21:45:20.230644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.230248    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"
Nov 03 21:45:20.236836 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.236814    5244 scope.go:122] "RemoveContainer" containerID="1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726"
Nov 03 21:45:20.239507 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:20.239485    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726\": container with ID starting with 1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726 not found: ID does not exist" containerID="1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726"
Nov 03 21:45:20.239683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.239660    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726"} err="failed to get container status \"1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726\": rpc error: code = NotFound desc = could not find container \"1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726\": container with ID starting with 1d108324a72fa3454869d5f9fd87a92bfc28dfacecef58a86a70f39686d40726 not found: ID does not exist"
Nov 03 21:45:20.239768 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.239758    5244 scope.go:122] "RemoveContainer" containerID="7e62ccd24a69446f2493233f96ea418f2d41685370a027618c895e0a56a35063"
Nov 03 21:45:20.254113 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.254088    5244 scope.go:122] "RemoveContainer" containerID="52db33aa8a66ba8f2d0fe7195d93577bdd3afb609f574e420a590d32ceca81b3"
Nov 03 21:45:20.383437 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.383401    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"]
Nov 03 21:45:20.387453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.387428    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-a1931b-29895705-sd5rz"]
Nov 03 21:45:20.495156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.495068    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"]
Nov 03 21:45:20.499025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.498994    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-85d7ff-7d45d69769-lb9wb"]
Nov 03 21:45:20.606253 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.606214    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"]
Nov 03 21:45:20.609781 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:20.609755    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-85d7ff-29895705-xtk25"]
Nov 03 21:45:21.135925 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:21.135889    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" path="/var/lib/kubelet/pods/2974da9b-a2fd-4290-b8fe-99baf4106285/volumes"
Nov 03 21:45:21.136416 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:21.136400    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961af80d-53aa-43f4-b164-d331e017aea4" path="/var/lib/kubelet/pods/961af80d-53aa-43f4-b164-d331e017aea4/volumes"
Nov 03 21:45:21.136808 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:21.136792    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" path="/var/lib/kubelet/pods/a656bb03-0a72-497c-9610-4a709fb63eb9/volumes"
Nov 03 21:45:22.436452 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:22.436415    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:45:36.493955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:45:36.493859    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:45:52.439178 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:45:52.439136    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:46:11.626544 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.626506    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"]
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.626980    5244 cpu_manager.go:414] "containerMap: removing container" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.626993    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627013    5244 cpu_manager.go:414] "containerMap: removing container" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627018    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627029    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627035    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627041    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627046    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627060    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627064    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627074    5244 cpu_manager.go:414] "containerMap: removing container" podUID="961af80d-53aa-43f4-b164-d331e017aea4" containerName="maas-api"
Nov 03 21:46:11.627101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627078    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961af80d-53aa-43f4-b164-d331e017aea4" containerName="maas-api"
Nov 03 21:46:11.627714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627139    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.627714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627148    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:11.627714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627154    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="961af80d-53aa-43f4-b164-d331e017aea4" containerName="maas-api"
Nov 03 21:46:11.627714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627162    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:11.627714 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.627169    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:11.630489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.630470    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.682142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.682099    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-87e1cf-openshift-default-dockercfg-28lnl\""
Nov 03 21:46:11.728445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.728406    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.728685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.728451    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.728685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.728672    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.728795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.728723    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.728850 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.728816    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.728959 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.728939    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.729033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.729016    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.729126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.729107    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59cx\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.729197 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.729180    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.761932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.761888    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"]
Nov 03 21:46:11.830607 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830571    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830607 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830608    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830657    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830675    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830700    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830746    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830827    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.830867 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830863    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59cx\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.831171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.830924    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.831171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.831032    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.831171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.831052    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.831456 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.831430    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.831551 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.831469    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.831681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.831666    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.833236 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.833219    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.833340 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.833309    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.888994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.888918    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.888994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.888980    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59cx\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx\") pod \"e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") " pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:11.944378 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:11.944319    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:12.379272 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.379238    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"]
Nov 03 21:46:12.380955 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:46:12.380926    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e519501_99b3_474a_b49d_71b43984f860.slice/crio-95d77c0ab5bd097e8ff998d147d3828a1932dd9695cde352c8022334ffa94e5c WatchSource:0}: Error finding container 95d77c0ab5bd097e8ff998d147d3828a1932dd9695cde352c8022334ffa94e5c: Status 404 returned error can't find the container with id 95d77c0ab5bd097e8ff998d147d3828a1932dd9695cde352c8022334ffa94e5c
Nov 03 21:46:12.383684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.383650    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:46:12.383758 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.383723    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:46:12.383758 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.383751    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:46:12.573121 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.573078    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" event={"ID":"5e519501-99b3-474a-b49d-71b43984f860","Type":"ContainerStarted","Data":"f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7"}
Nov 03 21:46:12.573346 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.573192    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" event={"ID":"5e519501-99b3-474a-b49d-71b43984f860","Type":"ContainerStarted","Data":"95d77c0ab5bd097e8ff998d147d3828a1932dd9695cde352c8022334ffa94e5c"}
Nov 03 21:46:12.693315 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.693242    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podStartSLOduration=1.6932280849999999 podStartE2EDuration="1.693228085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:12.692510368 +0000 UTC m=+1843.676506972" watchObservedRunningTime="2026-11-03 21:46:12.693228085 +0000 UTC m=+1843.677224643"
Nov 03 21:46:12.950540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:12.950437    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:13.948100 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:13.948059    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 21:46:13.948757 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:13.948720    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 21:46:14.958900 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:14.958862    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 21:46:14.959410 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:14.958922    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 21:46:15.956483 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:15.956445    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 21:46:15.956698 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:15.956500    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 21:46:16.950357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:16.947804    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 21:46:16.950357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:16.947875    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 21:46:17.955357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:17.954704    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 21:46:17.955357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:17.954769    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 21:46:18.958230 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:18.958198    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:18.958896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:18.958828    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:18.959562 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:18.959543    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 21:46:22.438115 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:46:22.438070    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:46:29.788551 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:29.788507    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:46:44.154172 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.154139    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"]
Nov 03 21:46:44.154838 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.154788    5244 cpu_manager.go:414] "containerMap: removing container" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:44.154838 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.154805    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:44.154912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.154884    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="a656bb03-0a72-497c-9610-4a709fb63eb9" containerName="cleanup"
Nov 03 21:46:44.154912 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.154898    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2974da9b-a2fd-4290-b8fe-99baf4106285" containerName="cleanup"
Nov 03 21:46:44.158265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.158247    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.197385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.197313    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zfn\" (UniqueName: \"kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.197585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.197494    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.197643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.197617    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.197679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.197648    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.209778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.209752    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3c6597-serving-cert\""
Nov 03 21:46:44.261568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.261522    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"]
Nov 03 21:46:44.298818 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.298765    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.299011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.298823    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.299011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.298988    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zfn\" (UniqueName: \"kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.299150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.299094    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.299491 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:46:44.299260    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-serving-cert: secret "maas-api-e2e-api-b-3c6597-serving-cert" not found
Nov 03 21:46:44.299491 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:46:44.299364    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls podName:8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec nodeName:}" failed. No retries permitted until 2026-11-03 21:46:44.799313484 +0000 UTC m=+1875.783310040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls") pod "maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" (UID: "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec") : secret "maas-api-e2e-api-b-3c6597-serving-cert" not found
Nov 03 21:46:44.299683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.299624    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.302814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.302777    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.357844 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.357809    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zfn\" (UniqueName: \"kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.818772 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.818725    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:44.821389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:44.821361    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\") pod \"maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:45.071973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:45.071877    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:45.752764 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:45.752727    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"]
Nov 03 21:46:45.796549 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:46:45.796520    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7d5eda_a111_4446_8ac1_e23f1f2dd6ec.slice/crio-5d555c41655d0ed6586dc0299e6d93cb2451c0ac9b83fb2683b81c15209ea757 WatchSource:0}: Error finding container 5d555c41655d0ed6586dc0299e6d93cb2451c0ac9b83fb2683b81c15209ea757: Status 404 returned error can't find the container with id 5d555c41655d0ed6586dc0299e6d93cb2451c0ac9b83fb2683b81c15209ea757
Nov 03 21:46:45.797940 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:45.797923    5244 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:46:45.811730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:45.811701    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" event={"ID":"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec","Type":"ContainerStarted","Data":"5d555c41655d0ed6586dc0299e6d93cb2451c0ac9b83fb2683b81c15209ea757"}
Nov 03 21:46:46.818676 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:46.818637    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" event={"ID":"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec","Type":"ContainerStarted","Data":"b48a54725b554e9be3797c6443a8f0ecd154a9e7e335d07982da4a30217d715e"}
Nov 03 21:46:46.819120 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:46.818719    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:46.934074 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:46.934019    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" podStartSLOduration=2.126079393 podStartE2EDuration="2.934003938s" totalImagesPullingTime="807.924545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:46:46.932598017 +0000 UTC m=+1877.916594587" watchObservedRunningTime="2026-11-03 21:46:46.934003938 +0000 UTC m=+1877.918000498"
Nov 03 21:46:49.386199 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:49.386151    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"]
Nov 03 21:46:49.386820 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:49.386452    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" containerID="cri-o://f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd" gracePeriod=30
Nov 03 21:46:51.297054 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:51.297018    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:46:51.297436 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:51.297108    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:46:52.437903 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:46:52.437864    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:46:52.830890 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:52.830809    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:46:54.642578 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.642547    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:46:54.657584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657516    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657553    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657584    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657609    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657682    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657702    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657762    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657787    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657793    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:54.657854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.657842    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4\") pod \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\" (UID: \"6f12c21e-ca76-447f-9f0b-415b809f5b8c\") "
Nov 03 21:46:54.658195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658018    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:54.658195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658052    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:54.658195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658064    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:46:54.658195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658074    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:54.658452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658381    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.658452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658403    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.658452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658420    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.658452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658435    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.658452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.658449    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.660364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.660323    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:54.660719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.660699    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:46:54.660771 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.660710    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:46:54.660841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.660828    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4" pod "6f12c21e-ca76-447f-9f0b-415b809f5b8c" (UID: "6f12c21e-ca76-447f-9f0b-415b809f5b8c"). InnerVolumeSpecName "kube-api-access-msxg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:46:54.759477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.759436    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.759477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.759474    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.759477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.759487    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f12c21e-ca76-447f-9f0b-415b809f5b8c-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.759723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.759496    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxg4\" (UniqueName: \"kubernetes.io/projected/6f12c21e-ca76-447f-9f0b-415b809f5b8c-kube-api-access-msxg4\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:46:54.904927 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.904888    5244 generic.go:356] "Generic (PLEG): container finished" podID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerID="f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd" exitCode=0
Nov 03 21:46:54.905109 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.905077    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" event={"ID":"6f12c21e-ca76-447f-9f0b-415b809f5b8c","Type":"ContainerDied","Data":"f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd"}
Nov 03 21:46:54.905185 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.905170    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v" event={"ID":"6f12c21e-ca76-447f-9f0b-415b809f5b8c","Type":"ContainerDied","Data":"34c1fa546feb591d23106ee88c57a8524c13b91bc715671cdf6350e01d776658"}
Nov 03 21:46:54.905225 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.905207    5244 scope.go:122] "RemoveContainer" containerID="f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd"
Nov 03 21:46:54.905420 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.905401    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"
Nov 03 21:46:54.917585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.917565    5244 scope.go:122] "RemoveContainer" containerID="f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd"
Nov 03 21:46:54.917899 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:46:54.917882    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd\": container with ID starting with f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd not found: ID does not exist" containerID="f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd"
Nov 03 21:46:54.917976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:54.917912    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd"} err="failed to get container status \"f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd\": rpc error: code = NotFound desc = could not find container \"f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd\": container with ID starting with f8782571e27b95b848f97d6156d4eac5939db7550517e317e07f1463cdd600dd not found: ID does not exist"
Nov 03 21:46:55.068071 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:55.068038    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"]
Nov 03 21:46:55.071642 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:55.071607    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-s9f9v"]
Nov 03 21:46:55.136295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:46:55.136258    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" path="/var/lib/kubelet/pods/6f12c21e-ca76-447f-9f0b-415b809f5b8c/volumes"
Nov 03 21:47:10.451642 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:10.451557    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:47:22.436837 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:47:22.436800    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:47:23.920408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:23.920361    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"]
Nov 03 21:47:23.921040 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:23.921017    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy"
Nov 03 21:47:23.921101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:23.921044    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy"
Nov 03 21:47:23.921221 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:23.921205    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f12c21e-ca76-447f-9f0b-415b809f5b8c" containerName="istio-proxy"
Nov 03 21:47:23.924638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:23.924611    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:23.976474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:23.976441    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-204763-openshift-default-dockercfg-6s5mr\""
Nov 03 21:47:24.026099 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026061    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026109    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026161    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026204    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp97\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026238    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026267    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026299    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026345    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.026469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.026419    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.029038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.029008    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"]
Nov 03 21:47:24.127457 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.127408    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.127764 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.127734    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.127904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.127786    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.127904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.127816    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.127904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.127893    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.128070 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.128022    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.128070 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.128054    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.128179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.128115    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.128179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.128159    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp97\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.128930 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.128903    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.129186 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.129164    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.129850 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.129443    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.129959 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.129859    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.130790 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.130759    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.136076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.136046    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.137024 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.137000    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.185915 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.185883    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp97\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.186120 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.186042    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token\") pod \"e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") " pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.239192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.239154    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:24.671202 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.671096    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"]
Nov 03 21:47:24.713505 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:47:24.713472    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode145f8d4_9961_4400_9e8c_587b7084e5d5.slice/crio-e06886020e7f67b39de30903e91442755996aeb0f9c353bb87f76deb020cb0d6 WatchSource:0}: Error finding container e06886020e7f67b39de30903e91442755996aeb0f9c353bb87f76deb020cb0d6: Status 404 returned error can't find the container with id e06886020e7f67b39de30903e91442755996aeb0f9c353bb87f76deb020cb0d6
Nov 03 21:47:24.716965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.716917    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:47:24.717063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.717022    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:47:24.717110 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:24.717071    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:47:25.136156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:25.136124    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" event={"ID":"e145f8d4-9961-4400-9e8c-587b7084e5d5","Type":"ContainerStarted","Data":"0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"}
Nov 03 21:47:25.136564 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:25.136175    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" event={"ID":"e145f8d4-9961-4400-9e8c-587b7084e5d5","Type":"ContainerStarted","Data":"e06886020e7f67b39de30903e91442755996aeb0f9c353bb87f76deb020cb0d6"}
Nov 03 21:47:25.239498 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:25.239466    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:25.245357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:25.245301    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:25.253218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:25.253164    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podStartSLOduration=2.253148491 podStartE2EDuration="2.253148491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:47:25.250164698 +0000 UTC m=+1916.234161257" watchObservedRunningTime="2026-11-03 21:47:25.253148491 +0000 UTC m=+1916.237145169"
Nov 03 21:47:26.138857 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:26.138824    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:26.140068 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:26.140041    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:47:45.382138 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:47:45.382099    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:47:52.437296 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:47:52.437249    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:48:03.184354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.179907    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"]
Nov 03 21:48:03.184354 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.180205    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" containerID="cri-o://fad81bc23f969c6c46974fe749748ab6a12cff6945dbf77a507a48c8170c11f9" gracePeriod=30
Nov 03 21:48:03.344574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.344542    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"]
Nov 03 21:48:03.348154 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.348131    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.405352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.402709    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:48:03.405352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.403550    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:48:03.424353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.414002    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"]
Nov 03 21:48:03.439146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.439052    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.439558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.439534    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.439926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.439906    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5kg\" (UniqueName: \"kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.542021 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.541992    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.542204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.542073    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.542204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.542136    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5kg\" (UniqueName: \"kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.542773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.542748    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.544683 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.544657    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.624196 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.624162    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5kg\" (UniqueName: \"kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg\") pod \"maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:03.680794 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:03.680757    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:04.149411 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:04.149307    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"]
Nov 03 21:48:04.153736 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:48:04.153694    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a9b890_48ae_49a2_a45c_14bfa8856e67.slice/crio-0f8e5398e1e8a3b6f1dd54aec8ba3dd4151cc5cd64d5969726398db8d4080193 WatchSource:0}: Error finding container 0f8e5398e1e8a3b6f1dd54aec8ba3dd4151cc5cd64d5969726398db8d4080193: Status 404 returned error can't find the container with id 0f8e5398e1e8a3b6f1dd54aec8ba3dd4151cc5cd64d5969726398db8d4080193
Nov 03 21:48:04.389015 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:04.388969    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5" event={"ID":"d0a9b890-48ae-49a2-a45c-14bfa8856e67","Type":"ContainerStarted","Data":"0f8e5398e1e8a3b6f1dd54aec8ba3dd4151cc5cd64d5969726398db8d4080193"}
Nov 03 21:48:05.396153 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:05.396058    5244 generic.go:356] "Generic (PLEG): container finished" podID="d0a9b890-48ae-49a2-a45c-14bfa8856e67" containerID="5f39a9681fafbb5f1dfaea987d753830b8f24c138ecc8c27e61652610a870f4b" exitCode=0
Nov 03 21:48:05.396153 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:05.396125    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5" event={"ID":"d0a9b890-48ae-49a2-a45c-14bfa8856e67","Type":"ContainerDied","Data":"5f39a9681fafbb5f1dfaea987d753830b8f24c138ecc8c27e61652610a870f4b"}
Nov 03 21:48:05.433828 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:05.433789    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:48:05.433994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:05.433846    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:48:06.562190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.562159    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:06.681009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.680924    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg\" (UniqueName: \"kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg\") pod \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") "
Nov 03 21:48:06.681009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.680973    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca\") pod \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") "
Nov 03 21:48:06.681267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.681043    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca\") pod \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\" (UID: \"d0a9b890-48ae-49a2-a45c-14bfa8856e67\") "
Nov 03 21:48:06.681489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.681467    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca" pod "d0a9b890-48ae-49a2-a45c-14bfa8856e67" (UID: "d0a9b890-48ae-49a2-a45c-14bfa8856e67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:06.681603 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.681583    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:06.683296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.683268    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg" pod "d0a9b890-48ae-49a2-a45c-14bfa8856e67" (UID: "d0a9b890-48ae-49a2-a45c-14bfa8856e67"). InnerVolumeSpecName "kube-api-access-bx5kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:06.683296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.683274    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca" pod "d0a9b890-48ae-49a2-a45c-14bfa8856e67" (UID: "d0a9b890-48ae-49a2-a45c-14bfa8856e67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:48:06.782643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.782607    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5kg\" (UniqueName: \"kubernetes.io/projected/d0a9b890-48ae-49a2-a45c-14bfa8856e67-kube-api-access-bx5kg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:06.782643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:06.782639    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a9b890-48ae-49a2-a45c-14bfa8856e67-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:07.409349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:07.409258    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"
Nov 03 21:48:07.409349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:07.409286    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5" event={"ID":"d0a9b890-48ae-49a2-a45c-14bfa8856e67","Type":"ContainerDied","Data":"0f8e5398e1e8a3b6f1dd54aec8ba3dd4151cc5cd64d5969726398db8d4080193"}
Nov 03 21:48:07.409349 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:07.409350    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8e5398e1e8a3b6f1dd54aec8ba3dd4151cc5cd64d5969726398db8d4080193"
Nov 03 21:48:08.424387 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.424302    5244 generic.go:356] "Generic (PLEG): container finished" podID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerID="fad81bc23f969c6c46974fe749748ab6a12cff6945dbf77a507a48c8170c11f9" exitCode=0
Nov 03 21:48:08.424807 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.424400    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" event={"ID":"d5067398-2831-4257-b8a0-35c66bd9b3d4","Type":"ContainerDied","Data":"fad81bc23f969c6c46974fe749748ab6a12cff6945dbf77a507a48c8170c11f9"}
Nov 03 21:48:08.454393 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.454367    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:48:08.604317 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604231    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604356    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604378    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604400    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604463    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604512    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604528    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604581    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604620    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs\") pod \"d5067398-2831-4257-b8a0-35c66bd9b3d4\" (UID: \"d5067398-2831-4257-b8a0-35c66bd9b3d4\") "
Nov 03 21:48:08.604863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604684    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:08.604863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604843    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:08.605241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604883    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:08.605241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.604987    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:08.605241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.605076    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:08.605241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.605206    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.605241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.605223    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.605241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.605241    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.605620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.605259    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5067398-2831-4257-b8a0-35c66bd9b3d4-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.605620 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.605269    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.606900 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.606873    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:48:08.607009 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.606903    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:08.607218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.607192    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "kube-api-access-2trsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:08.607218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.607194    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy" pod "d5067398-2831-4257-b8a0-35c66bd9b3d4" (UID: "d5067398-2831-4257-b8a0-35c66bd9b3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:08.706580 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.706535    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.706580 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.706575    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.706580 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.706586    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trsk\" (UniqueName: \"kubernetes.io/projected/d5067398-2831-4257-b8a0-35c66bd9b3d4-kube-api-access-2trsk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:08.706840 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:08.706600    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5067398-2831-4257-b8a0-35c66bd9b3d4-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:09.431822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.431796    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"
Nov 03 21:48:09.432290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.431814    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m" event={"ID":"d5067398-2831-4257-b8a0-35c66bd9b3d4","Type":"ContainerDied","Data":"0df7aab64401317ce67b6db2b50dfe05e9e37946db283ce09a4e81ac824f084f"}
Nov 03 21:48:09.432290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.431886    5244 scope.go:122] "RemoveContainer" containerID="fad81bc23f969c6c46974fe749748ab6a12cff6945dbf77a507a48c8170c11f9"
Nov 03 21:48:09.535276 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.535244    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"]
Nov 03 21:48:09.535484 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.535448    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" containerID="cri-o://e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557" gracePeriod=30
Nov 03 21:48:09.646163 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.646128    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"]
Nov 03 21:48:09.696114 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:09.695994    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-69f7m"]
Nov 03 21:48:11.136195 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.136162    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" path="/var/lib/kubelet/pods/d5067398-2831-4257-b8a0-35c66bd9b3d4/volumes"
Nov 03 21:48:11.278709 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.278670    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"]
Nov 03 21:48:11.280677 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.280649    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d0a9b890-48ae-49a2-a45c-14bfa8856e67" containerName="revoke-keys"
Nov 03 21:48:11.280857 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.280841    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a9b890-48ae-49a2-a45c-14bfa8856e67" containerName="revoke-keys"
Nov 03 21:48:11.281033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.281019    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy"
Nov 03 21:48:11.281126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.281112    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy"
Nov 03 21:48:11.281365 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.281350    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5067398-2831-4257-b8a0-35c66bd9b3d4" containerName="istio-proxy"
Nov 03 21:48:11.281476 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.281464    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a9b890-48ae-49a2-a45c-14bfa8856e67" containerName="revoke-keys"
Nov 03 21:48:11.303156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.303125    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.338355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.329105    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"]
Nov 03 21:48:11.366284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.366252    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-62cf5c-dockercfg-lr578\""
Nov 03 21:48:11.415113 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.415027    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-62cf5c\""
Nov 03 21:48:11.430956 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.430921    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"]
Nov 03 21:48:11.434608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.434589    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.463976 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.463944    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7l67\" (UniqueName: \"kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67\") pod \"payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.464167 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.464000    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.493563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.493524    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"]
Nov 03 21:48:11.565210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.565173    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5pb\" (UniqueName: \"kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.565462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.565248    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l67\" (UniqueName: \"kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67\") pod \"payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.565462 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.565406    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.565615 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.565506    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.566104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.566086    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.623035 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.623003    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l67\" (UniqueName: \"kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67\") pod \"payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.667291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.667208    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.667470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.667308    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5pb\" (UniqueName: \"kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.667984 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.667965    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.694989 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.694940    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:11.724992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.724958    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5pb\" (UniqueName: \"kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb\") pod \"payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:11.750085 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:11.750029    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:12.073767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.073735    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"]
Nov 03 21:48:12.076253 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:48:12.076227    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fdad14_0675_4faa_9b07_429880c95090.slice/crio-20e4a86d9e8a3eafd99eb90d2f144c7bb11ab8458b096e2eb39753c50a5d2ae1 WatchSource:0}: Error finding container 20e4a86d9e8a3eafd99eb90d2f144c7bb11ab8458b096e2eb39753c50a5d2ae1: Status 404 returned error can't find the container with id 20e4a86d9e8a3eafd99eb90d2f144c7bb11ab8458b096e2eb39753c50a5d2ae1
Nov 03 21:48:12.186745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.186713    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"]
Nov 03 21:48:12.189513 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:48:12.189471    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea6de24_9086_4419_9046_1f221d5a7a8e.slice/crio-9297391c9b4a6cfdfdc81b19dccec224ff459af9468b276c328376165283491c WatchSource:0}: Error finding container 9297391c9b4a6cfdfdc81b19dccec224ff459af9468b276c328376165283491c: Status 404 returned error can't find the container with id 9297391c9b4a6cfdfdc81b19dccec224ff459af9468b276c328376165283491c
Nov 03 21:48:12.322119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.322077    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:48:12.322258 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.322133    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:48:12.450129 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.450096    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls" event={"ID":"1ea6de24-9086-4419-9046-1f221d5a7a8e","Type":"ContainerStarted","Data":"3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0"}
Nov 03 21:48:12.450383 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.450160    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls" event={"ID":"1ea6de24-9086-4419-9046-1f221d5a7a8e","Type":"ContainerStarted","Data":"9297391c9b4a6cfdfdc81b19dccec224ff459af9468b276c328376165283491c"}
Nov 03 21:48:12.450383 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.450185    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:12.452048 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.452020    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f" event={"ID":"f0fdad14-0675-4faa-9b07-429880c95090","Type":"ContainerStarted","Data":"8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24"}
Nov 03 21:48:12.452168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.452076    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f" event={"ID":"f0fdad14-0675-4faa-9b07-429880c95090","Type":"ContainerStarted","Data":"20e4a86d9e8a3eafd99eb90d2f144c7bb11ab8458b096e2eb39753c50a5d2ae1"}
Nov 03 21:48:12.452168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.452130    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:12.564955 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.564898    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls" podStartSLOduration=1.564879122 podStartE2EDuration="1.564879122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:12.563136533 +0000 UTC m=+1963.547133114" watchObservedRunningTime="2026-11-03 21:48:12.564879122 +0000 UTC m=+1963.548875680"
Nov 03 21:48:12.690662 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:12.690581    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f" podStartSLOduration=1.690557088 podStartE2EDuration="1.690557088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:12.687769643 +0000 UTC m=+1963.671766199" watchObservedRunningTime="2026-11-03 21:48:12.690557088 +0000 UTC m=+1963.674553644"
Nov 03 21:48:14.880723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.880699    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:48:14.934372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934308    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934375    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934593 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934411    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934593 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934431    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934593 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934488    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934740 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934620    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934740 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934659    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934740 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934671    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:14.934740 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934684    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:14.934740 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934710    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934971 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934757    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert\") pod \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\" (UID: \"7d99735e-8eea-446b-9cd9-4cb2a716ef52\") "
Nov 03 21:48:14.934971 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934845    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:14.934971 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.934879    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:14.935151 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.935132    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:48:14.935453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.935429    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:14.935568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.935457    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:14.935568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.935473    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:14.935568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.935487    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:14.935568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.935506    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:14.936896 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.936877    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:48:14.937027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.936999    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:14.937360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.937322    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "kube-api-access-s85tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:48:14.937428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:14.937416    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy" pod "7d99735e-8eea-446b-9cd9-4cb2a716ef52" (UID: "7d99735e-8eea-446b-9cd9-4cb2a716ef52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:48:15.037222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.037170    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85tx\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-kube-api-access-s85tx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:15.037222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.037215    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:15.037222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.037230    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:15.037575 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.037243    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d99735e-8eea-446b-9cd9-4cb2a716ef52-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:48:15.469728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.469696    5244 generic.go:356] "Generic (PLEG): container finished" podID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerID="e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557" exitCode=0
Nov 03 21:48:15.469939 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.469758    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"
Nov 03 21:48:15.469939 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.469780    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" event={"ID":"7d99735e-8eea-446b-9cd9-4cb2a716ef52","Type":"ContainerDied","Data":"e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557"}
Nov 03 21:48:15.469939 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.469842    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g" event={"ID":"7d99735e-8eea-446b-9cd9-4cb2a716ef52","Type":"ContainerDied","Data":"76097fa0677e24b49f9a99fc2424a767733e5f59855591779618931336c376bd"}
Nov 03 21:48:15.469939 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.469873    5244 scope.go:122] "RemoveContainer" containerID="e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557"
Nov 03 21:48:15.479119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.479102    5244 scope.go:122] "RemoveContainer" containerID="e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557"
Nov 03 21:48:15.479440 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:48:15.479413    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557\": container with ID starting with e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557 not found: ID does not exist" containerID="e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557"
Nov 03 21:48:15.479528 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.479454    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557"} err="failed to get container status \"e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557\": rpc error: code = NotFound desc = could not find container \"e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557\": container with ID starting with e5e71fcd142427ecb95610d5953c9e79ccbee7b295df6b7bff44d54d9ffe7557 not found: ID does not exist"
Nov 03 21:48:15.633400 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.633359    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"]
Nov 03 21:48:15.682759 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:15.682728    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-s669g"]
Nov 03 21:48:17.136390 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:17.136353    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" path="/var/lib/kubelet/pods/7d99735e-8eea-446b-9cd9-4cb2a716ef52/volumes"
Nov 03 21:48:19.930408 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:19.930373    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:48:22.436664 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:48:22.436622    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:48:23.460317 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:23.460287    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:48:23.460752 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:23.460405    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:48:26.953575 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:26.953538    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"]
Nov 03 21:48:26.954033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:26.954021    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy"
Nov 03 21:48:26.954082 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:26.954035    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy"
Nov 03 21:48:26.954120 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:26.954103    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d99735e-8eea-446b-9cd9-4cb2a716ef52" containerName="istio-proxy"
Nov 03 21:48:26.957363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:26.957347    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.009486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.009457    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1c3bed\""
Nov 03 21:48:27.058852 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.058821    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1c3bed-dockercfg-wgt4h\""
Nov 03 21:48:27.061169 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.061141    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"]
Nov 03 21:48:27.094192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.094158    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbphd\" (UniqueName: \"kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd\") pod \"payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.094375 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.094235    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.196074 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.196035    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbphd\" (UniqueName: \"kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd\") pod \"payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.196280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.196131    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.196871 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.196847    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.253082 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.253006    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbphd\" (UniqueName: \"kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd\") pod \"payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.276965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.276930    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:27.651773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:27.651749    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"]
Nov 03 21:48:27.653479 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:48:27.653453    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99f73bc_8f0e_4778_bed8_cfae7ed9dce1.slice/crio-8e4185086d69d8b4b930adad238349fcfa5b4a350d3246780c1ebbb4bb633155 WatchSource:0}: Error finding container 8e4185086d69d8b4b930adad238349fcfa5b4a350d3246780c1ebbb4bb633155: Status 404 returned error can't find the container with id 8e4185086d69d8b4b930adad238349fcfa5b4a350d3246780c1ebbb4bb633155
Nov 03 21:48:28.554253 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:28.554222    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6" event={"ID":"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1","Type":"ContainerStarted","Data":"8e4185086d69d8b4b930adad238349fcfa5b4a350d3246780c1ebbb4bb633155"}
Nov 03 21:48:30.567675 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:30.567636    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6" event={"ID":"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1","Type":"ContainerStarted","Data":"9ad10660629c55a0b723f5349ec1d9a725bf32ebced41107f630952dfc2cf82b"}
Nov 03 21:48:30.568111 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:30.567814    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:30.681204 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:30.681135    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6" podStartSLOduration=2.684464045 podStartE2EDuration="4.681116296s" totalImagesPullingTime="1.996652251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:30.681111997 +0000 UTC m=+1981.665108570" watchObservedRunningTime="2026-11-03 21:48:30.681116296 +0000 UTC m=+1981.665112854"
Nov 03 21:48:39.148177 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.148084    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"]
Nov 03 21:48:39.152766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.152743    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.226574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.226546    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ae2f6f-openshift-default-dockercfg-ttp4c\""
Nov 03 21:48:39.249459 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249420    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.249655 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249569    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.249655 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249615    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.249759 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249660    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.249842 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249809    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvjv\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.249977 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249846    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.249977 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249938    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.250106 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.249973    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.250106 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.250072    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.257989 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.257953    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"]
Nov 03 21:48:39.357400 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.357366    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.364775 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.359136    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.365118 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.365092    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.365372 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.365353    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.365547 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.365531    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvjv\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.365686 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.365661    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.365840 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.365825    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.365994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.365968    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.366136 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.360711    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.366212 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.358133    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.378363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.369876    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.378363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.369926    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.378363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.370166    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.378363 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.371100    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.385033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.380975    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.385033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.384231    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.422394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.422302    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.428763 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.428731    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvjv\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv\") pod \"e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") " pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.469877 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.469824    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:39.944388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.944352    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"]
Nov 03 21:48:39.946363 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:48:39.946310    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6292679d_0d43_426e_bf2f_9ce73a137393.slice/crio-58039ccb570f90312eb6136b7a65a7a8b40913fb3cb55daf9092ce794a663144 WatchSource:0}: Error finding container 58039ccb570f90312eb6136b7a65a7a8b40913fb3cb55daf9092ce794a663144: Status 404 returned error can't find the container with id 58039ccb570f90312eb6136b7a65a7a8b40913fb3cb55daf9092ce794a663144
Nov 03 21:48:39.949168 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.949125    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:48:39.949273 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.949220    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:48:39.949343 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:39.949277    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:48:40.622792 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:40.622752    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" event={"ID":"6292679d-0d43-426e-bf2f-9ce73a137393","Type":"ContainerStarted","Data":"563e3e555e7836fa6af413486e723363bbf812096a2069f78ac979543f3df746"}
Nov 03 21:48:40.623215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:40.622814    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" event={"ID":"6292679d-0d43-426e-bf2f-9ce73a137393","Type":"ContainerStarted","Data":"58039ccb570f90312eb6136b7a65a7a8b40913fb3cb55daf9092ce794a663144"}
Nov 03 21:48:40.839219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:40.839167    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podStartSLOduration=1.83915297 podStartE2EDuration="1.83915297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:40.789107746 +0000 UTC m=+1991.773104330" watchObservedRunningTime="2026-11-03 21:48:40.83915297 +0000 UTC m=+1991.823149528"
Nov 03 21:48:41.470909 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:41.470865    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:41.476936 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:41.476907    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:41.575488 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:41.575457    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:48:41.630074 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:41.630043    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:41.631365 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:41.631314    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:48:48.140187 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.140146    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"]
Nov 03 21:48:48.145973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.145947    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.199684 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.199644    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"]
Nov 03 21:48:48.263192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.263152    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.263695 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.263668    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtqx\" (UniqueName: \"kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.284280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.264086    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ae2f6f\""
Nov 03 21:48:48.284550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.264220    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ae2f6f-dockercfg-ldlgq\""
Nov 03 21:48:48.384576 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.384547    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtqx\" (UniqueName: \"kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.384901 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.384884    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.385860 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.385834    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.440861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.440825    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtqx\" (UniqueName: \"kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.461147 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.461118    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:48.842644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:48.842612    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"]
Nov 03 21:48:48.844729 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:48:48.844700    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abf1311_b409_4ec0_bf10_39853bdb5156.slice/crio-1917f3af28b7cadb0673b4f1a15501bf90c66718d6d1d3985b0d993ab5e38341 WatchSource:0}: Error finding container 1917f3af28b7cadb0673b4f1a15501bf90c66718d6d1d3985b0d993ab5e38341: Status 404 returned error can't find the container with id 1917f3af28b7cadb0673b4f1a15501bf90c66718d6d1d3985b0d993ab5e38341
Nov 03 21:48:49.699870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:49.699833    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6" event={"ID":"3abf1311-b409-4ec0-bf10-39853bdb5156","Type":"ContainerStarted","Data":"66e87c4e870fd7c52ccee97e832031b768123772ea063113d86f0dcc1784436a"}
Nov 03 21:48:49.700278 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:49.699892    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6" event={"ID":"3abf1311-b409-4ec0-bf10-39853bdb5156","Type":"ContainerStarted","Data":"1917f3af28b7cadb0673b4f1a15501bf90c66718d6d1d3985b0d993ab5e38341"}
Nov 03 21:48:49.700278 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:49.699997    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:48:49.814564 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:49.814501    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6" podStartSLOduration=1.814486607 podStartE2EDuration="1.814486607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:48:49.811952619 +0000 UTC m=+2000.795949196" watchObservedRunningTime="2026-11-03 21:48:49.814486607 +0000 UTC m=+2000.798483193"
Nov 03 21:48:52.437249 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:48:52.437206    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:48:54.914423 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:48:54.914386    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:49:00.710405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:00.710364    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:49:02.453949 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.453908    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"]
Nov 03 21:49:02.459809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.459774    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.516626 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.516579    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"]
Nov 03 21:49:02.562468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.562378    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"openshift-service-ca.crt\""
Nov 03 21:49:02.562468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.562391    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"kube-root-ca.crt\""
Nov 03 21:49:02.562468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.562429    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"default-dockercfg-77mh2\""
Nov 03 21:49:02.562804 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.562497    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-87e1cf\"/\"tenantcfg-model-87e1cf-kserve-self-signed-certs\""
Nov 03 21:49:02.607280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.606576    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.607280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.606621    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.607280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.606786    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g555\" (UniqueName: \"kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.607280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.606870    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.607280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.606915    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.607280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.606954    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.708542 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.708446    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g555\" (UniqueName: \"kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.708542 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.708522    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.708789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.708550    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.708789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.708579    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.708789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.708624    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.708789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.708647    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.709227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.709040    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.709227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.709051    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.709227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.709155    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.711825 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.711803    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.712115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.712095    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.765682 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.765641    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g555\" (UniqueName: \"kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555\") pod \"tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") " pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:02.774889 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:02.774845    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:03.169572 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:03.169538    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"]
Nov 03 21:49:03.171724 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:49:03.171693    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a208866_745d_4596_81d3_5426946a2e88.slice/crio-dc4d709eaa857d22912b1a11d9e568ea7095159bff53f542c245b9260108b06b WatchSource:0}: Error finding container dc4d709eaa857d22912b1a11d9e568ea7095159bff53f542c245b9260108b06b: Status 404 returned error can't find the container with id dc4d709eaa857d22912b1a11d9e568ea7095159bff53f542c245b9260108b06b
Nov 03 21:49:03.802397 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:03.802354    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" event={"ID":"9a208866-745d-4596-81d3-5426946a2e88","Type":"ContainerStarted","Data":"04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15"}
Nov 03 21:49:03.802807 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:03.802408    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" event={"ID":"9a208866-745d-4596-81d3-5426946a2e88","Type":"ContainerStarted","Data":"dc4d709eaa857d22912b1a11d9e568ea7095159bff53f542c245b9260108b06b"}
Nov 03 21:49:03.918477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:03.918421    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" podStartSLOduration=1.9184068029999999 podStartE2EDuration="1.918406803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:03.916052684 +0000 UTC m=+2014.900049244" watchObservedRunningTime="2026-11-03 21:49:03.918406803 +0000 UTC m=+2014.902403362"
Nov 03 21:49:04.775203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:04.775163    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:05.815854 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:05.815822    5244 generic.go:356] "Generic (PLEG): container finished" podID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerID="0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9" exitCode=0
Nov 03 21:49:05.816308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:05.815893    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" event={"ID":"e145f8d4-9961-4400-9e8c-587b7084e5d5","Type":"ContainerDied","Data":"0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"}
Nov 03 21:49:05.816438 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:05.816420    5244 scope.go:122] "RemoveContainer" containerID="0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"
Nov 03 21:49:05.817875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:05.817848    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:49:05.817935 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:05.817921    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:49:05.817974 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:05.817960    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:49:06.829179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:06.829146    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" event={"ID":"e145f8d4-9961-4400-9e8c-587b7084e5d5","Type":"ContainerStarted","Data":"fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af"}
Nov 03 21:49:07.241729 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.241695    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:49:07.248487 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.248453    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 21:49:07.248779 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.248744    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 21:49:07.785262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.785217    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"]
Nov 03 21:49:07.789090 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.789072    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:07.855412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.855371    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3c0bdb-dockercfg-gs9fm\""
Nov 03 21:49:07.857371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.856067    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3c0bdb\""
Nov 03 21:49:07.859830 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.859799    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"]
Nov 03 21:49:07.905614 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.905569    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mll\" (UniqueName: \"kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:07.905821 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:07.905724    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.006621 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.006576    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mll\" (UniqueName: \"kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.006815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.006638    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.007385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.007363    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.064348 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.064244    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mll\" (UniqueName: \"kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll\") pod \"payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.102501 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.102465    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.254357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.243769    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 21:49:08.254357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.243835    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 21:49:08.501214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.501184    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"]
Nov 03 21:49:08.503615 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:49:08.503584    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe191d0_6151_42e4_8fb4_bccd7d013e9a.slice/crio-ea8a275d665fce38eb6417c10ff4511a415a03cb8acb0c30c83bdfb8c9309555 WatchSource:0}: Error finding container ea8a275d665fce38eb6417c10ff4511a415a03cb8acb0c30c83bdfb8c9309555: Status 404 returned error can't find the container with id ea8a275d665fce38eb6417c10ff4511a415a03cb8acb0c30c83bdfb8c9309555
Nov 03 21:49:08.615608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.615571    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"]
Nov 03 21:49:08.619584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.619556    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.686144 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.686104    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"]
Nov 03 21:49:08.717747 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.717712    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.717960 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.717787    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs29v\" (UniqueName: \"kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v\") pod \"payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.818767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.818677    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.818767 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.818734    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs29v\" (UniqueName: \"kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v\") pod \"payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.819525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.819497    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.869863 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.869828    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf" event={"ID":"4fe191d0-6151-42e4-8fb4-bccd7d013e9a","Type":"ContainerStarted","Data":"68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a"}
Nov 03 21:49:08.870375 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.869891    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:08.870375 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.869907    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf" event={"ID":"4fe191d0-6151-42e4-8fb4-bccd7d013e9a","Type":"ContainerStarted","Data":"ea8a275d665fce38eb6417c10ff4511a415a03cb8acb0c30c83bdfb8c9309555"}
Nov 03 21:49:08.875906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.875885    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs29v\" (UniqueName: \"kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v\") pod \"payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.957758 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.957712    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:08.983973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:08.983914    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf" podStartSLOduration=1.983896155 podStartE2EDuration="1.983896155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:08.981745865 +0000 UTC m=+2019.965742449" watchObservedRunningTime="2026-11-03 21:49:08.983896155 +0000 UTC m=+2019.967892713"
Nov 03 21:49:09.257529 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.257138    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 21:49:09.257529 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.257205    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 21:49:09.360385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.358546    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"]
Nov 03 21:49:09.365681 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:49:09.365646    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8996f32b_aac0_4266_a050_340e9dbea2de.slice/crio-f9528632e17b12475e2a0c184fa1877e8c14bd44b9a78571ec5a86dba9f69716 WatchSource:0}: Error finding container f9528632e17b12475e2a0c184fa1877e8c14bd44b9a78571ec5a86dba9f69716: Status 404 returned error can't find the container with id f9528632e17b12475e2a0c184fa1877e8c14bd44b9a78571ec5a86dba9f69716
Nov 03 21:49:09.878609 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.878568    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf" event={"ID":"8996f32b-aac0-4266-a050-340e9dbea2de","Type":"ContainerStarted","Data":"535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe"}
Nov 03 21:49:09.879076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.878628    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf" event={"ID":"8996f32b-aac0-4266-a050-340e9dbea2de","Type":"ContainerStarted","Data":"f9528632e17b12475e2a0c184fa1877e8c14bd44b9a78571ec5a86dba9f69716"}
Nov 03 21:49:09.879076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.878657    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:09.993140 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:09.993048    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf" podStartSLOduration=1.9930281829999998 podStartE2EDuration="1.993028183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:09.990836763 +0000 UTC m=+2020.974833340" watchObservedRunningTime="2026-11-03 21:49:09.993028183 +0000 UTC m=+2020.977024742"
Nov 03 21:49:10.247221 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:10.246747    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 21:49:10.247221 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:10.246811    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 21:49:11.166982 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:11.166952    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:49:11.246811 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:11.246771    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 21:49:11.247118 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:11.247088    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 21:49:12.244191 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:12.244159    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:49:12.245246 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:12.245223    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:49:12.775290 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:12.775259    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:12.788187 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:12.788160    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:12.909927 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:12.909891    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:49:19.881033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:19.881004    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:49:20.886470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:20.886436    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:49:22.437498 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:49:22.437448    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:49:22.926574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:22.926472    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:49:44.998519 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:44.998471    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"]
Nov 03 21:49:45.004048 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.004023    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.057487 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.057447    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3940f8\""
Nov 03 21:49:45.107139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.107107    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3940f8-dockercfg-8lx56\""
Nov 03 21:49:45.109501 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.109480    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"]
Nov 03 21:49:45.129833 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.129794    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.130013 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.129965    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55k7\" (UniqueName: \"kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.231633 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.231596    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.231817 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.231707    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55k7\" (UniqueName: \"kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.232295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.232273    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.298098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.298022    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55k7\" (UniqueName: \"kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7\") pod \"payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.320432 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.320398    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:45.695532 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.695507    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"]
Nov 03 21:49:45.697402 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:49:45.697365    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137991db_2ac5_4ca6_8e0b_7701035514ba.slice/crio-31cecda5888fdf7d7bfb8772981be3dc658c13f3fc15d45ebba58c0df8367246 WatchSource:0}: Error finding container 31cecda5888fdf7d7bfb8772981be3dc658c13f3fc15d45ebba58c0df8367246: Status 404 returned error can't find the container with id 31cecda5888fdf7d7bfb8772981be3dc658c13f3fc15d45ebba58c0df8367246
Nov 03 21:49:45.786771 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.786734    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"]
Nov 03 21:49:45.790826 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.790800    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:45.839685 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.839648    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wd6\" (UniqueName: \"kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6\") pod \"payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:45.839887 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.839713    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:45.849219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.849186    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"]
Nov 03 21:49:45.941091 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.941046    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wd6\" (UniqueName: \"kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6\") pod \"payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:45.941307 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.941131    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:45.941930 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:45.941902    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:46.000776 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.000704    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wd6\" (UniqueName: \"kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6\") pod \"payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:46.105566 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.105536    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:46.109193 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.109168    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j" event={"ID":"137991db-2ac5-4ca6-8e0b-7701035514ba","Type":"ContainerStarted","Data":"4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4"}
Nov 03 21:49:46.109311 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.109216    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j" event={"ID":"137991db-2ac5-4ca6-8e0b-7701035514ba","Type":"ContainerStarted","Data":"31cecda5888fdf7d7bfb8772981be3dc658c13f3fc15d45ebba58c0df8367246"}
Nov 03 21:49:46.109371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.109309    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:46.225608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.225539    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j" podStartSLOduration=2.225520723 podStartE2EDuration="2.225520723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:46.223310552 +0000 UTC m=+2057.207307129" watchObservedRunningTime="2026-11-03 21:49:46.225520723 +0000 UTC m=+2057.209517283"
Nov 03 21:49:46.499632 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:46.499604    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"]
Nov 03 21:49:46.501570 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:49:46.501541    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd73fece_1e0e_4244_ab0f_1f9b14786472.slice/crio-de8933bf54fd2e303815ed8571e31699e89c9506e773189fefc093ccd5449b0c WatchSource:0}: Error finding container de8933bf54fd2e303815ed8571e31699e89c9506e773189fefc093ccd5449b0c: Status 404 returned error can't find the container with id de8933bf54fd2e303815ed8571e31699e89c9506e773189fefc093ccd5449b0c
Nov 03 21:49:47.118197 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:47.118160    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4" event={"ID":"cd73fece-1e0e-4244-ab0f-1f9b14786472","Type":"ContainerStarted","Data":"7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030"}
Nov 03 21:49:47.118657 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:47.118223    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4" event={"ID":"cd73fece-1e0e-4244-ab0f-1f9b14786472","Type":"ContainerStarted","Data":"de8933bf54fd2e303815ed8571e31699e89c9506e773189fefc093ccd5449b0c"}
Nov 03 21:49:47.232262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:47.232196    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4" podStartSLOduration=2.232176188 podStartE2EDuration="2.232176188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:47.231135941 +0000 UTC m=+2058.215132500" watchObservedRunningTime="2026-11-03 21:49:47.232176188 +0000 UTC m=+2058.216172753"
Nov 03 21:49:48.123869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:48.123836    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:49:52.437621 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:49:52.437577    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:49:57.120034 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:57.119997    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:49:57.555558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:57.555513    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"]
Nov 03 21:49:57.557820 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:57.557793    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1931b-mxwh5"]
Nov 03 21:49:57.863782 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:57.863693    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:49:59.140366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:59.140311    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a9b890-48ae-49a2-a45c-14bfa8856e67" path="/var/lib/kubelet/pods/d0a9b890-48ae-49a2-a45c-14bfa8856e67/volumes"
Nov 03 21:49:59.140861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:49:59.140741    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:50:05.333243 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.333195    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"]
Nov 03 21:50:05.337585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.337557    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.384495 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.384452    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.384694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.384600    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.384754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.384726    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvg5x\" (UniqueName: \"kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.442018 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.441985    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:50:05.442018 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.442024    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:50:05.446081 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.446048    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"]
Nov 03 21:50:05.486173 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.486136    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvg5x\" (UniqueName: \"kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.486429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.486256    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.486429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.486373    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.487101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.487075    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.489134 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.489102    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.582353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.565567    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvg5x\" (UniqueName: \"kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x\") pod \"maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:05.662351 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:05.662246    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:06.044265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:06.044242    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"]
Nov 03 21:50:06.046145 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:50:06.046112    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5ce2cf_6b5b_4ab5_8ce3_1898fe4564df.slice/crio-a82d4a15957f0273ab565a7bdde185ed45cadc71dd6efbf58c8cdabb2e49ef9c WatchSource:0}: Error finding container a82d4a15957f0273ab565a7bdde185ed45cadc71dd6efbf58c8cdabb2e49ef9c: Status 404 returned error can't find the container with id a82d4a15957f0273ab565a7bdde185ed45cadc71dd6efbf58c8cdabb2e49ef9c
Nov 03 21:50:06.284508 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:06.284465    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc" event={"ID":"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df","Type":"ContainerStarted","Data":"a82d4a15957f0273ab565a7bdde185ed45cadc71dd6efbf58c8cdabb2e49ef9c"}
Nov 03 21:50:07.292520 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:07.292426    5244 generic.go:356] "Generic (PLEG): container finished" podID="0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" containerID="fe96b3b38176af177a995fcd836f0cc3145cb58ec8f97b36c149230a436d0fbc" exitCode=0
Nov 03 21:50:07.292520 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:07.292490    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc" event={"ID":"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df","Type":"ContainerDied","Data":"fe96b3b38176af177a995fcd836f0cc3145cb58ec8f97b36c149230a436d0fbc"}
Nov 03 21:50:08.517310 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.517284    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:08.644088 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.643996    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca\") pod \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") "
Nov 03 21:50:08.644258 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.644236    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca\") pod \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") "
Nov 03 21:50:08.645052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.644275    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x\" (UniqueName: \"kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x\") pod \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\" (UID: \"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df\") "
Nov 03 21:50:08.645052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.644643    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca" pod "0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" (UID: "0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:50:08.645052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.644791    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:08.646833 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.646799    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca" pod "0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" (UID: "0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:50:08.646940 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.646891    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x" pod "0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" (UID: "0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df"). InnerVolumeSpecName "kube-api-access-mvg5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:50:08.746000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.745960    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:08.746000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:08.745994    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvg5x\" (UniqueName: \"kubernetes.io/projected/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df-kube-api-access-mvg5x\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:09.330249 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:09.330214    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"
Nov 03 21:50:09.330249 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:09.330238    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc" event={"ID":"0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df","Type":"ContainerDied","Data":"a82d4a15957f0273ab565a7bdde185ed45cadc71dd6efbf58c8cdabb2e49ef9c"}
Nov 03 21:50:09.330489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:09.330294    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a82d4a15957f0273ab565a7bdde185ed45cadc71dd6efbf58c8cdabb2e49ef9c"
Nov 03 21:50:22.464451 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:50:22.464399    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:50:30.524823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:30.524731    5244 generic.go:356] "Generic (PLEG): container finished" podID="6292679d-0d43-426e-bf2f-9ce73a137393" containerID="563e3e555e7836fa6af413486e723363bbf812096a2069f78ac979543f3df746" exitCode=0
Nov 03 21:50:30.524823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:30.524804    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" event={"ID":"6292679d-0d43-426e-bf2f-9ce73a137393","Type":"ContainerDied","Data":"563e3e555e7836fa6af413486e723363bbf812096a2069f78ac979543f3df746"}
Nov 03 21:50:30.525383 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:30.525259    5244 scope.go:122] "RemoveContainer" containerID="563e3e555e7836fa6af413486e723363bbf812096a2069f78ac979543f3df746"
Nov 03 21:50:30.526951 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:30.526915    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:50:30.527042 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:30.526989    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:50:30.527042 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:30.527020    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 21:50:31.559182 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:31.559149    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" event={"ID":"6292679d-0d43-426e-bf2f-9ce73a137393","Type":"ContainerStarted","Data":"6b4340e1dc4c68b56971a0003ec6f612178afc9e84b437527d20dfcef0dd8e15"}
Nov 03 21:50:32.470162 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:32.470124    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:50:32.470371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:32.470287    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 21:50:32.470456 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:32.470376    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 21:50:33.470799 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:33.470766    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 21:50:33.471293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:33.470826    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 21:50:34.470301 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:34.470263    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 21:50:34.470506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:34.470339    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 21:50:35.470174 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:35.470138    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 21:50:35.470623 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:35.470194    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 21:50:36.470643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:36.470598    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 21:50:36.471063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:36.470672    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 21:50:36.972267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:36.972229    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"]
Nov 03 21:50:37.076556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:37.076516    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"]
Nov 03 21:50:37.475445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:37.475413    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:50:37.475866 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:37.475732    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:50:37.476479 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:37.476456    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:50:38.060319 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.060284    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"]
Nov 03 21:50:38.060855 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.060840    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" containerName="revoke-keys"
Nov 03 21:50:38.060906 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.060855    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" containerName="revoke-keys"
Nov 03 21:50:38.060966 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.060955    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" containerName="revoke-keys"
Nov 03 21:50:38.063171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.063154    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.119213 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.119185    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:50:38.119469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.119206    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:50:38.122975 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.122946    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"]
Nov 03 21:50:38.155464 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.155431    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.155638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.155516    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kdlg\" (UniqueName: \"kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.155688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.155645    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.256552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.256509    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.256729 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.256609    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kdlg\" (UniqueName: \"kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.256729 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.256663    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.257340 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.257306    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.259081 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.259058    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.313098 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.313022    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kdlg\" (UniqueName: \"kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg\") pod \"maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.376158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.376129    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:38.759088 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:38.758715    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"]
Nov 03 21:50:38.763515 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:50:38.763381    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode291ed97_5c7e_4341_a108_a6242ca2dbd0.slice/crio-a3dc0e428df7217246f7c3c01021909c14785d69afc1533db200b0c4864fd889 WatchSource:0}: Error finding container a3dc0e428df7217246f7c3c01021909c14785d69afc1533db200b0c4864fd889: Status 404 returned error can't find the container with id a3dc0e428df7217246f7c3c01021909c14785d69afc1533db200b0c4864fd889
Nov 03 21:50:39.617402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:39.609878    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx" event={"ID":"e291ed97-5c7e-4341-a108-a6242ca2dbd0","Type":"ContainerStarted","Data":"35a326ee23c15386be12dec748cb73269994925faead05db4b7152bc1b9c7c40"}
Nov 03 21:50:39.617402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:39.610302    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx" event={"ID":"e291ed97-5c7e-4341-a108-a6242ca2dbd0","Type":"ContainerStarted","Data":"a3dc0e428df7217246f7c3c01021909c14785d69afc1533db200b0c4864fd889"}
Nov 03 21:50:39.724743 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:39.724687    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx" podStartSLOduration=1.724672929 podStartE2EDuration="1.724672929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:50:39.722738846 +0000 UTC m=+2110.706735417" watchObservedRunningTime="2026-11-03 21:50:39.724672929 +0000 UTC m=+2110.708669488"
Nov 03 21:50:40.617524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:40.617486    5244 generic.go:356] "Generic (PLEG): container finished" podID="e291ed97-5c7e-4341-a108-a6242ca2dbd0" containerID="35a326ee23c15386be12dec748cb73269994925faead05db4b7152bc1b9c7c40" exitCode=0
Nov 03 21:50:40.617940 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:40.617574    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx" event={"ID":"e291ed97-5c7e-4341-a108-a6242ca2dbd0","Type":"ContainerDied","Data":"35a326ee23c15386be12dec748cb73269994925faead05db4b7152bc1b9c7c40"}
Nov 03 21:50:41.773485 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.773459    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:41.848473 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.848434    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca\") pod \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") "
Nov 03 21:50:41.848690 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.848593    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca\") pod \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") "
Nov 03 21:50:41.848765 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.848697    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg\" (UniqueName: \"kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg\") pod \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\" (UID: \"e291ed97-5c7e-4341-a108-a6242ca2dbd0\") "
Nov 03 21:50:41.848988 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.848947    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca" pod "e291ed97-5c7e-4341-a108-a6242ca2dbd0" (UID: "e291ed97-5c7e-4341-a108-a6242ca2dbd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:50:41.849163 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.849147    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:41.850861 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.850839    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca" pod "e291ed97-5c7e-4341-a108-a6242ca2dbd0" (UID: "e291ed97-5c7e-4341-a108-a6242ca2dbd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:50:41.851044 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.851023    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg" pod "e291ed97-5c7e-4341-a108-a6242ca2dbd0" (UID: "e291ed97-5c7e-4341-a108-a6242ca2dbd0"). InnerVolumeSpecName "kube-api-access-2kdlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:50:41.950064 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.950028    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e291ed97-5c7e-4341-a108-a6242ca2dbd0-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:41.950064 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.950057    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kdlg\" (UniqueName: \"kubernetes.io/projected/e291ed97-5c7e-4341-a108-a6242ca2dbd0-kube-api-access-2kdlg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:41.973614 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:41.973572    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f" podUID="f0fdad14-0675-4faa-9b07-429880c95090" containerName="payload-processing" containerID="cri-o://8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24" gracePeriod=25
Nov 03 21:50:42.077206 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.077147    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls" podUID="1ea6de24-9086-4419-9046-1f221d5a7a8e" containerName="payload-pre-processing" containerID="cri-o://3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0" gracePeriod=25
Nov 03 21:50:42.230011 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.229985    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:50:42.354773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.354739    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume\") pod \"f0fdad14-0675-4faa-9b07-429880c95090\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") "
Nov 03 21:50:42.354953 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.354827    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67\" (UniqueName: \"kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67\") pod \"f0fdad14-0675-4faa-9b07-429880c95090\" (UID: \"f0fdad14-0675-4faa-9b07-429880c95090\") "
Nov 03 21:50:42.355179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.355155    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume" pod "f0fdad14-0675-4faa-9b07-429880c95090" (UID: "f0fdad14-0675-4faa-9b07-429880c95090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:50:42.355406 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.355364    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0fdad14-0675-4faa-9b07-429880c95090-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:42.357156 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.357133    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67" pod "f0fdad14-0675-4faa-9b07-429880c95090" (UID: "f0fdad14-0675-4faa-9b07-429880c95090"). InnerVolumeSpecName "kube-api-access-t7l67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:50:42.368205 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.368181    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:50:42.456134 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.456101    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb\" (UniqueName: \"kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb\") pod \"1ea6de24-9086-4419-9046-1f221d5a7a8e\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") "
Nov 03 21:50:42.456392 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.456207    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume\") pod \"1ea6de24-9086-4419-9046-1f221d5a7a8e\" (UID: \"1ea6de24-9086-4419-9046-1f221d5a7a8e\") "
Nov 03 21:50:42.456694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.456668    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume" pod "1ea6de24-9086-4419-9046-1f221d5a7a8e" (UID: "1ea6de24-9086-4419-9046-1f221d5a7a8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:50:42.456873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.456858    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7l67\" (UniqueName: \"kubernetes.io/projected/f0fdad14-0675-4faa-9b07-429880c95090-kube-api-access-t7l67\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:42.456916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.456876    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea6de24-9086-4419-9046-1f221d5a7a8e-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:42.458552 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.458526    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb" pod "1ea6de24-9086-4419-9046-1f221d5a7a8e" (UID: "1ea6de24-9086-4419-9046-1f221d5a7a8e"). InnerVolumeSpecName "kube-api-access-xc5pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:50:42.558128 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.558032    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc5pb\" (UniqueName: \"kubernetes.io/projected/1ea6de24-9086-4419-9046-1f221d5a7a8e-kube-api-access-xc5pb\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:50:42.632822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.632789    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"
Nov 03 21:50:42.632986 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.632822    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx" event={"ID":"e291ed97-5c7e-4341-a108-a6242ca2dbd0","Type":"ContainerDied","Data":"a3dc0e428df7217246f7c3c01021909c14785d69afc1533db200b0c4864fd889"}
Nov 03 21:50:42.632986 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.632869    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3dc0e428df7217246f7c3c01021909c14785d69afc1533db200b0c4864fd889"
Nov 03 21:50:42.634222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.634194    5244 generic.go:356] "Generic (PLEG): container finished" podID="1ea6de24-9086-4419-9046-1f221d5a7a8e" containerID="3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0" exitCode=0
Nov 03 21:50:42.634366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.634229    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"
Nov 03 21:50:42.634366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.634271    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls" event={"ID":"1ea6de24-9086-4419-9046-1f221d5a7a8e","Type":"ContainerDied","Data":"3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0"}
Nov 03 21:50:42.634366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.634319    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls" event={"ID":"1ea6de24-9086-4419-9046-1f221d5a7a8e","Type":"ContainerDied","Data":"9297391c9b4a6cfdfdc81b19dccec224ff459af9468b276c328376165283491c"}
Nov 03 21:50:42.634366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.634363    5244 scope.go:122] "RemoveContainer" containerID="3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0"
Nov 03 21:50:42.636794 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.636773    5244 generic.go:356] "Generic (PLEG): container finished" podID="f0fdad14-0675-4faa-9b07-429880c95090" containerID="8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24" exitCode=0
Nov 03 21:50:42.636926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.636822    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"
Nov 03 21:50:42.636926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.636837    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f" event={"ID":"f0fdad14-0675-4faa-9b07-429880c95090","Type":"ContainerDied","Data":"8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24"}
Nov 03 21:50:42.636926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.636868    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f" event={"ID":"f0fdad14-0675-4faa-9b07-429880c95090","Type":"ContainerDied","Data":"20e4a86d9e8a3eafd99eb90d2f144c7bb11ab8458b096e2eb39753c50a5d2ae1"}
Nov 03 21:50:42.646291 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.646276    5244 scope.go:122] "RemoveContainer" containerID="3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0"
Nov 03 21:50:42.646735 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:50:42.646713    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0\": container with ID starting with 3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0 not found: ID does not exist" containerID="3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0"
Nov 03 21:50:42.646815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.646745    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0"} err="failed to get container status \"3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0\": rpc error: code = NotFound desc = could not find container \"3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0\": container with ID starting with 3c3c65feebe360c26d155c2d095c1abf70f9f163ec72f31668a5657208debfc0 not found: ID does not exist"
Nov 03 21:50:42.646815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.646762    5244 scope.go:122] "RemoveContainer" containerID="8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24"
Nov 03 21:50:42.656312 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.656295    5244 scope.go:122] "RemoveContainer" containerID="8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24"
Nov 03 21:50:42.656596 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:50:42.656581    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24\": container with ID starting with 8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24 not found: ID does not exist" containerID="8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24"
Nov 03 21:50:42.656643 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.656603    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24"} err="failed to get container status \"8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24\": rpc error: code = NotFound desc = could not find container \"8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24\": container with ID starting with 8635886fbe57d7a7c8fdecf2ffb2a1e8f84c74e3f781566979e4a08b2b0eef24 not found: ID does not exist"
Nov 03 21:50:42.796521 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.796474    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"]
Nov 03 21:50:42.798690 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.798670    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-62cf5c-77cf969bd-rjl7f"]
Nov 03 21:50:42.909468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.909187    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"]
Nov 03 21:50:42.912431 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:42.912404    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-62cf5c-54c7754587-9hhls"]
Nov 03 21:50:43.143389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:43.141464    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea6de24-9086-4419-9046-1f221d5a7a8e" path="/var/lib/kubelet/pods/1ea6de24-9086-4419-9046-1f221d5a7a8e/volumes"
Nov 03 21:50:43.143389 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:43.142131    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0fdad14-0675-4faa-9b07-429880c95090" path="/var/lib/kubelet/pods/f0fdad14-0675-4faa-9b07-429880c95090/volumes"
Nov 03 21:50:47.858094 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:50:47.858056    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:50:52.437795 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:50:52.437742    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:51:22.332215 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:22.332165    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"]
Nov 03 21:51:22.332865 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:22.332477    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy" containerID="cri-o://0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8" gracePeriod=30
Nov 03 21:51:22.334422 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:22.334397    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"]
Nov 03 21:51:22.334597 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:22.334578    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy" containerID="cri-o://bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0" gracePeriod=30
Nov 03 21:51:22.403492 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:22.403449    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:51:22.455227 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:22.455188    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:51:22.742308 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:22.742261    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"]
Nov 03 21:51:26.949399 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:26.949354    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"]
Nov 03 21:51:26.949878 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:26.949588    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" containerID="cri-o://dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd" gracePeriod=30
Nov 03 21:51:27.603875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.603798    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:51:27.647585 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.647557    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:51:27.681664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681630    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.681882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681681    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.681882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681704    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.681882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681728    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.681882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681748    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.681882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681777    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.681882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681836    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681914    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681940    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.681976    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682034    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682102    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682144    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682175    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data\") pod \"28fc277d-6aae-4767-9706-c22e71ea9c02\" (UID: \"28fc277d-6aae-4767-9706-c22e71ea9c02\") "
Nov 03 21:51:27.682214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682208    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.682809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682244    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.682809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682292    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.682809 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682365    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket\") pod \"305d0fe0-cc53-4967-9425-175ffd72720f\" (UID: \"305d0fe0-cc53-4967-9425-175ffd72720f\") "
Nov 03 21:51:27.684435 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682029    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682049    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682418    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682620    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.682965    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.683077    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.683480    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.683957    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.684090    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:27.684571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.684396    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.685613 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.685588    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:27.685737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.685622    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "kube-api-access-s4gqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:27.685737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.685685    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "kube-api-access-jt5bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:27.685737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.685717    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:27.685902 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.685795    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.685977 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.685962    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:27.686277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.686264    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token" pod "305d0fe0-cc53-4967-9425-175ffd72720f" (UID: "305d0fe0-cc53-4967-9425-175ffd72720f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:27.686745 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.686727    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo" pod "28fc277d-6aae-4767-9706-c22e71ea9c02" (UID: "28fc277d-6aae-4767-9706-c22e71ea9c02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:27.743394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.743351    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6" podUID="3abf1311-b409-4ec0-bf10-39853bdb5156" containerName="payload-pre-processing" containerID="cri-o://66e87c4e870fd7c52ccee97e832031b768123772ea063113d86f0dcc1784436a" gracePeriod=25
Nov 03 21:51:27.783754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783717    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783750    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783761    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783783    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5bh\" (UniqueName: \"kubernetes.io/projected/305d0fe0-cc53-4967-9425-175ffd72720f-kube-api-access-jt5bh\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783798    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783812    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783824    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783837    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783850    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783863    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783875    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783887    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28fc277d-6aae-4767-9706-c22e71ea9c02-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783900    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305d0fe0-cc53-4967-9425-175ffd72720f-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783912    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4gqf\" (UniqueName: \"kubernetes.io/projected/28fc277d-6aae-4767-9706-c22e71ea9c02-kube-api-access-s4gqf\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783926    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783938    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305d0fe0-cc53-4967-9425-175ffd72720f-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783949    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28fc277d-6aae-4767-9706-c22e71ea9c02-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.783973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.783960    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305d0fe0-cc53-4967-9425-175ffd72720f-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:27.996060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.996025    5244 generic.go:356] "Generic (PLEG): container finished" podID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerID="bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0" exitCode=0
Nov 03 21:51:27.996613 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.996090    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"
Nov 03 21:51:27.996613 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.996127    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" event={"ID":"28fc277d-6aae-4767-9706-c22e71ea9c02","Type":"ContainerDied","Data":"bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0"}
Nov 03 21:51:27.996613 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.996179    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7" event={"ID":"28fc277d-6aae-4767-9706-c22e71ea9c02","Type":"ContainerDied","Data":"b742c26d41645a102d7de45d172ab5422be80daf5760d0801f96572f09ace590"}
Nov 03 21:51:27.996613 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.996212    5244 scope.go:122] "RemoveContainer" containerID="bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0"
Nov 03 21:51:27.999275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.999132    5244 generic.go:356] "Generic (PLEG): container finished" podID="305d0fe0-cc53-4967-9425-175ffd72720f" containerID="0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8" exitCode=0
Nov 03 21:51:27.999862 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.999756    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" event={"ID":"305d0fe0-cc53-4967-9425-175ffd72720f","Type":"ContainerDied","Data":"0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8"}
Nov 03 21:51:27.999862 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:27.999811    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd" event={"ID":"305d0fe0-cc53-4967-9425-175ffd72720f","Type":"ContainerDied","Data":"08c04a6efe563bcfef2a36b20ce055981316991d36d8b114d9e58769a01bf08f"}
Nov 03 21:51:28.001771 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.001733    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"
Nov 03 21:51:28.002243 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.002214    5244 generic.go:356] "Generic (PLEG): container finished" podID="3abf1311-b409-4ec0-bf10-39853bdb5156" containerID="66e87c4e870fd7c52ccee97e832031b768123772ea063113d86f0dcc1784436a" exitCode=0
Nov 03 21:51:28.002386 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.002285    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6" event={"ID":"3abf1311-b409-4ec0-bf10-39853bdb5156","Type":"ContainerDied","Data":"66e87c4e870fd7c52ccee97e832031b768123772ea063113d86f0dcc1784436a"}
Nov 03 21:51:28.011563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.011532    5244 scope.go:122] "RemoveContainer" containerID="bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0"
Nov 03 21:51:28.011904 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:28.011876    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0\": container with ID starting with bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0 not found: ID does not exist" containerID="bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0"
Nov 03 21:51:28.011978 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.011918    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0"} err="failed to get container status \"bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0\": rpc error: code = NotFound desc = could not find container \"bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0\": container with ID starting with bc0c7ada1955fe8b03f02be379393290c432b2c1ca750675828e0b4361b767b0 not found: ID does not exist"
Nov 03 21:51:28.011978 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.011958    5244 scope.go:122] "RemoveContainer" containerID="0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8"
Nov 03 21:51:28.025776 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.025751    5244 scope.go:122] "RemoveContainer" containerID="0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8"
Nov 03 21:51:28.026377 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:28.026317    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8\": container with ID starting with 0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8 not found: ID does not exist" containerID="0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8"
Nov 03 21:51:28.026502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.026388    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8"} err="failed to get container status \"0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8\": rpc error: code = NotFound desc = could not find container \"0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8\": container with ID starting with 0c98e0b8c7de5c1b178fb17c7c4a43121bf300895546dad60eb691a406a9bfd8 not found: ID does not exist"
Nov 03 21:51:28.027946 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.027923    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:51:28.087006 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.086975    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume\") pod \"3abf1311-b409-4ec0-bf10-39853bdb5156\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") "
Nov 03 21:51:28.087220 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.087107    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx\" (UniqueName: \"kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx\") pod \"3abf1311-b409-4ec0-bf10-39853bdb5156\" (UID: \"3abf1311-b409-4ec0-bf10-39853bdb5156\") "
Nov 03 21:51:28.087477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.087443    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume" pod "3abf1311-b409-4ec0-bf10-39853bdb5156" (UID: "3abf1311-b409-4ec0-bf10-39853bdb5156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:28.089660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.089640    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx" pod "3abf1311-b409-4ec0-bf10-39853bdb5156" (UID: "3abf1311-b409-4ec0-bf10-39853bdb5156"). InnerVolumeSpecName "kube-api-access-wrtqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:28.161723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.161641    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"]
Nov 03 21:51:28.162116 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.162092    5244 patch_prober.go:29] interesting pod/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:51:28.162218 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.162141    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:51:28.188510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.188476    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3abf1311-b409-4ec0-bf10-39853bdb5156-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:28.188510 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.188504    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtqx\" (UniqueName: \"kubernetes.io/projected/3abf1311-b409-4ec0-bf10-39853bdb5156-kube-api-access-wrtqx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:28.192322 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.192296    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-thkfd"]
Nov 03 21:51:28.312851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.312816    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"]
Nov 03 21:51:28.324921 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:28.324895    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-22pp7"]
Nov 03 21:51:29.012546 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.012506    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6" event={"ID":"3abf1311-b409-4ec0-bf10-39853bdb5156","Type":"ContainerDied","Data":"1917f3af28b7cadb0673b4f1a15501bf90c66718d6d1d3985b0d993ab5e38341"}
Nov 03 21:51:29.012546 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.012545    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"
Nov 03 21:51:29.013124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.012614    5244 scope.go:122] "RemoveContainer" containerID="66e87c4e870fd7c52ccee97e832031b768123772ea063113d86f0dcc1784436a"
Nov 03 21:51:29.137251 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.137220    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" path="/var/lib/kubelet/pods/28fc277d-6aae-4767-9706-c22e71ea9c02/volumes"
Nov 03 21:51:29.137807 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.137791    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" path="/var/lib/kubelet/pods/305d0fe0-cc53-4967-9425-175ffd72720f/volumes"
Nov 03 21:51:29.175924 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.175880    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"]
Nov 03 21:51:29.178566 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:29.178539    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-5b6b69c499-mjdg6"]
Nov 03 21:51:31.137005 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:31.136967    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abf1311-b409-4ec0-bf10-39853bdb5156" path="/var/lib/kubelet/pods/3abf1311-b409-4ec0-bf10-39853bdb5156/volumes"
Nov 03 21:51:32.198289 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.198260    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:51:32.345598 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345501    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345598 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345558    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345601    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345637    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345678    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345725    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345794    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345826    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.345851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345838    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:32.346213 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.345934    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket\") pod \"bb02b475-191c-4dc5-b815-33afa4f756b6\" (UID: \"bb02b475-191c-4dc5-b815-33afa4f756b6\") "
Nov 03 21:51:32.346213 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346029    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:32.346213 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346100    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:32.346509 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346467    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:32.346633 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346515    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:32.346777 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346757    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.346841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346784    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.346841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346799    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb02b475-191c-4dc5-b815-33afa4f756b6-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.346841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346813    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.346841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.346827    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.348165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.348136    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:51:32.348277 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.348259    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:32.349012 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.348993    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "kube-api-access-282kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:32.349160 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.349138    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo" pod "bb02b475-191c-4dc5-b815-33afa4f756b6" (UID: "bb02b475-191c-4dc5-b815-33afa4f756b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:51:32.447812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.447768    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282kx\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-kube-api-access-282kx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.447812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.447810    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.448038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.447825    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:32.448038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:32.447839    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb02b475-191c-4dc5-b815-33afa4f756b6-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.065270 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.065229    5244 generic.go:356] "Generic (PLEG): container finished" podID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerID="dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd" exitCode=0
Nov 03 21:51:33.066198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.065595    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"
Nov 03 21:51:33.068672 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.065616    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" event={"ID":"bb02b475-191c-4dc5-b815-33afa4f756b6","Type":"ContainerDied","Data":"dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd"}
Nov 03 21:51:33.068890 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.068871    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz" event={"ID":"bb02b475-191c-4dc5-b815-33afa4f756b6","Type":"ContainerDied","Data":"afeab1d48e7fa58dc3b4dca289facaaa08b004ef0ee9e6e088a60e7649029e07"}
Nov 03 21:51:33.069139 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.069075    5244 scope.go:122] "RemoveContainer" containerID="dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd"
Nov 03 21:51:33.099158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.099126    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"]
Nov 03 21:51:33.100644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.100110    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf" podUID="8996f32b-aac0-4266-a050-340e9dbea2de" containerName="payload-processing" containerID="cri-o://535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe" gracePeriod=30
Nov 03 21:51:33.134559 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.134538    5244 scope.go:122] "RemoveContainer" containerID="dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd"
Nov 03 21:51:33.134949 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:33.134899    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd\": container with ID starting with dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd not found: ID does not exist" containerID="dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd"
Nov 03 21:51:33.134999 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.134963    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd"} err="failed to get container status \"dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd\": rpc error: code = NotFound desc = could not find container \"dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd\": container with ID starting with dc66c7aa68b225918d79960828258ae26c2ecf4317d6a3c2340dceeb2169fafd not found: ID does not exist"
Nov 03 21:51:33.147006 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.146976    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"]
Nov 03 21:51:33.147212 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.147173    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf" podUID="4fe191d0-6151-42e4-8fb4-bccd7d013e9a" containerName="payload-pre-processing" containerID="cri-o://68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a" gracePeriod=30
Nov 03 21:51:33.246571 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.246538    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"]
Nov 03 21:51:33.252841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.252805    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f463c9-openshift-default-75547c9bfc-g6jgz"]
Nov 03 21:51:33.449467 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.449439    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:51:33.453231 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.453204    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:51:33.572680 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.572583    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v\" (UniqueName: \"kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v\") pod \"8996f32b-aac0-4266-a050-340e9dbea2de\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") "
Nov 03 21:51:33.572869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.572760    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume\") pod \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") "
Nov 03 21:51:33.572869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.572814    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll\" (UniqueName: \"kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll\") pod \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\" (UID: \"4fe191d0-6151-42e4-8fb4-bccd7d013e9a\") "
Nov 03 21:51:33.572985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.572946    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume\") pod \"8996f32b-aac0-4266-a050-340e9dbea2de\" (UID: \"8996f32b-aac0-4266-a050-340e9dbea2de\") "
Nov 03 21:51:33.573150 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.573118    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume" pod "4fe191d0-6151-42e4-8fb4-bccd7d013e9a" (UID: "4fe191d0-6151-42e4-8fb4-bccd7d013e9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:33.573486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.573297    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume" pod "8996f32b-aac0-4266-a050-340e9dbea2de" (UID: "8996f32b-aac0-4266-a050-340e9dbea2de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:33.573486 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.573324    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.575000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.574981    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v" pod "8996f32b-aac0-4266-a050-340e9dbea2de" (UID: "8996f32b-aac0-4266-a050-340e9dbea2de"). InnerVolumeSpecName "kube-api-access-xs29v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:33.575083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.575009    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll" pod "4fe191d0-6151-42e4-8fb4-bccd7d013e9a" (UID: "4fe191d0-6151-42e4-8fb4-bccd7d013e9a"). InnerVolumeSpecName "kube-api-access-k8mll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:33.676056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.674351    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8mll\" (UniqueName: \"kubernetes.io/projected/4fe191d0-6151-42e4-8fb4-bccd7d013e9a-kube-api-access-k8mll\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.676056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.674393    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8996f32b-aac0-4266-a050-340e9dbea2de-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.676056 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.674408    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs29v\" (UniqueName: \"kubernetes.io/projected/8996f32b-aac0-4266-a050-340e9dbea2de-kube-api-access-xs29v\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:33.821952 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.821916    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"]
Nov 03 21:51:33.822142 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:33.822116    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" podUID="8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" containerName="maas-api" containerID="cri-o://b48a54725b554e9be3797c6443a8f0ecd154a9e7e335d07982da4a30217d715e" gracePeriod=30
Nov 03 21:51:34.137158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.137031    5244 generic.go:356] "Generic (PLEG): container finished" podID="8996f32b-aac0-4266-a050-340e9dbea2de" containerID="535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe" exitCode=0
Nov 03 21:51:34.137402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.137183    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf" event={"ID":"8996f32b-aac0-4266-a050-340e9dbea2de","Type":"ContainerDied","Data":"535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe"}
Nov 03 21:51:34.137402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.137236    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf" event={"ID":"8996f32b-aac0-4266-a050-340e9dbea2de","Type":"ContainerDied","Data":"f9528632e17b12475e2a0c184fa1877e8c14bd44b9a78571ec5a86dba9f69716"}
Nov 03 21:51:34.137402 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.137268    5244 scope.go:122] "RemoveContainer" containerID="535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe"
Nov 03 21:51:34.137653 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.137418    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"
Nov 03 21:51:34.176060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.145688    5244 generic.go:356] "Generic (PLEG): container finished" podID="4fe191d0-6151-42e4-8fb4-bccd7d013e9a" containerID="68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a" exitCode=0
Nov 03 21:51:34.176060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.145878    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf" event={"ID":"4fe191d0-6151-42e4-8fb4-bccd7d013e9a","Type":"ContainerDied","Data":"68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a"}
Nov 03 21:51:34.176060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.145930    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf" event={"ID":"4fe191d0-6151-42e4-8fb4-bccd7d013e9a","Type":"ContainerDied","Data":"ea8a275d665fce38eb6417c10ff4511a415a03cb8acb0c30c83bdfb8c9309555"}
Nov 03 21:51:34.176060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.146005    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"
Nov 03 21:51:34.176060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.151278    5244 generic.go:356] "Generic (PLEG): container finished" podID="8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" containerID="b48a54725b554e9be3797c6443a8f0ecd154a9e7e335d07982da4a30217d715e" exitCode=0
Nov 03 21:51:34.176060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.151413    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" event={"ID":"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec","Type":"ContainerDied","Data":"b48a54725b554e9be3797c6443a8f0ecd154a9e7e335d07982da4a30217d715e"}
Nov 03 21:51:34.235563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.235431    5244 scope.go:122] "RemoveContainer" containerID="535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe"
Nov 03 21:51:34.236379 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:34.236179    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe\": container with ID starting with 535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe not found: ID does not exist" containerID="535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe"
Nov 03 21:51:34.236379 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.236224    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe"} err="failed to get container status \"535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe\": rpc error: code = NotFound desc = could not find container \"535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe\": container with ID starting with 535ddcf19065ee33aea368a9029907866b10acfdea9e1cb0a300d9e36dddfdfe not found: ID does not exist"
Nov 03 21:51:34.236379 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.236247    5244 scope.go:122] "RemoveContainer" containerID="68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a"
Nov 03 21:51:34.264850 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.264824    5244 scope.go:122] "RemoveContainer" containerID="68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a"
Nov 03 21:51:34.265254 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:34.265223    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a\": container with ID starting with 68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a not found: ID does not exist" containerID="68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a"
Nov 03 21:51:34.265314 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.265269    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a"} err="failed to get container status \"68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a\": rpc error: code = NotFound desc = could not find container \"68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a\": container with ID starting with 68ab88f7f18d24014e1813b53fddcf125c121a0094b76cf9127fd7ac28b84f3a not found: ID does not exist"
Nov 03 21:51:34.268464 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.268438    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:51:34.354356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.348509    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\") pod \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") "
Nov 03 21:51:34.354356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.348677    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs\") pod \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") "
Nov 03 21:51:34.354356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.348827    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn\" (UniqueName: \"kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn\") pod \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") "
Nov 03 21:51:34.354356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.348893    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca\") pod \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\" (UID: \"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec\") "
Nov 03 21:51:34.354356 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.349672    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca" pod "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" (UID: "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:34.368357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.360349    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"]
Nov 03 21:51:34.368357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.364021    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c0bdb-5659ccc886-cszjf"]
Nov 03 21:51:34.381992 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.381931    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls" pod "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" (UID: "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:34.382250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.382223    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn" pod "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" (UID: "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec"). InnerVolumeSpecName "kube-api-access-h6zfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:34.386635 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.386599    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs" pod "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" (UID: "8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:34.450806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.450770    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:34.450806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.450799    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6zfn\" (UniqueName: \"kubernetes.io/projected/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-kube-api-access-h6zfn\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:34.450806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.450811    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:34.451121 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.450820    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:34.471754 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.471715    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"]
Nov 03 21:51:34.476885 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.476854    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c0bdb-76c794bc6c-p74nf"]
Nov 03 21:51:34.984035 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.983996    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"]
Nov 03 21:51:34.986214 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:34.986181    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3c0bdb-w4zrx"]
Nov 03 21:51:35.136832 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.136794    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe191d0-6151-42e4-8fb4-bccd7d013e9a" path="/var/lib/kubelet/pods/4fe191d0-6151-42e4-8fb4-bccd7d013e9a/volumes"
Nov 03 21:51:35.137276 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.137255    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8996f32b-aac0-4266-a050-340e9dbea2de" path="/var/lib/kubelet/pods/8996f32b-aac0-4266-a050-340e9dbea2de/volumes"
Nov 03 21:51:35.137770 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.137753    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" path="/var/lib/kubelet/pods/bb02b475-191c-4dc5-b815-33afa4f756b6/volumes"
Nov 03 21:51:35.138441 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.138423    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e291ed97-5c7e-4341-a108-a6242ca2dbd0" path="/var/lib/kubelet/pods/e291ed97-5c7e-4341-a108-a6242ca2dbd0/volumes"
Nov 03 21:51:35.159815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.159787    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"
Nov 03 21:51:35.159965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.159812    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8" event={"ID":"8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec","Type":"ContainerDied","Data":"5d555c41655d0ed6586dc0299e6d93cb2451c0ac9b83fb2683b81c15209ea757"}
Nov 03 21:51:35.159965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.159876    5244 scope.go:122] "RemoveContainer" containerID="b48a54725b554e9be3797c6443a8f0ecd154a9e7e335d07982da4a30217d715e"
Nov 03 21:51:35.322375 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.322283    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"]
Nov 03 21:51:35.328370 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:35.328310    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3c6597-7479b9b86d-z7wx8"]
Nov 03 21:51:36.093081 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:36.093047    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"]
Nov 03 21:51:36.094868 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:36.094844    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c6597-gd9xc"]
Nov 03 21:51:37.135849 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:37.135772    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df" path="/var/lib/kubelet/pods/0e5ce2cf-6b5b-4ab5-8ce3-1898fe4564df/volumes"
Nov 03 21:51:37.136258 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:37.136243    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" path="/var/lib/kubelet/pods/8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec/volumes"
Nov 03 21:51:42.325276 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325236    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"]
Nov 03 21:51:42.325773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325743    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f0fdad14-0675-4faa-9b07-429880c95090" containerName="payload-processing"
Nov 03 21:51:42.325773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325762    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fdad14-0675-4faa-9b07-429880c95090" containerName="payload-processing"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325779    5244 cpu_manager.go:414] "containerMap: removing container" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325784    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325795    5244 cpu_manager.go:414] "containerMap: removing container" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325800    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325808    5244 cpu_manager.go:414] "containerMap: removing container" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325812    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325818    5244 cpu_manager.go:414] "containerMap: removing container" podUID="3abf1311-b409-4ec0-bf10-39853bdb5156" containerName="payload-pre-processing"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325822    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abf1311-b409-4ec0-bf10-39853bdb5156" containerName="payload-pre-processing"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325838    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1ea6de24-9086-4419-9046-1f221d5a7a8e" containerName="payload-pre-processing"
Nov 03 21:51:42.325845 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325846    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea6de24-9086-4419-9046-1f221d5a7a8e" containerName="payload-pre-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325855    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" containerName="maas-api"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325860    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" containerName="maas-api"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325874    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8996f32b-aac0-4266-a050-340e9dbea2de" containerName="payload-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325879    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8996f32b-aac0-4266-a050-340e9dbea2de" containerName="payload-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325886    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e291ed97-5c7e-4341-a108-a6242ca2dbd0" containerName="revoke-keys"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325890    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e291ed97-5c7e-4341-a108-a6242ca2dbd0" containerName="revoke-keys"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325899    5244 cpu_manager.go:414] "containerMap: removing container" podUID="4fe191d0-6151-42e4-8fb4-bccd7d013e9a" containerName="payload-pre-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325905    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe191d0-6151-42e4-8fb4-bccd7d013e9a" containerName="payload-pre-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325974    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b7d5eda-a111-4446-8ac1-e23f1f2dd6ec" containerName="maas-api"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325984    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8996f32b-aac0-4266-a050-340e9dbea2de" containerName="payload-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.325992    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e291ed97-5c7e-4341-a108-a6242ca2dbd0" containerName="revoke-keys"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326002    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe191d0-6151-42e4-8fb4-bccd7d013e9a" containerName="payload-pre-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326009    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fdad14-0675-4faa-9b07-429880c95090" containerName="payload-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326015    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="305d0fe0-cc53-4967-9425-175ffd72720f" containerName="istio-proxy"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326021    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fc277d-6aae-4767-9706-c22e71ea9c02" containerName="istio-proxy"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326027    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb02b475-191c-4dc5-b815-33afa4f756b6" containerName="istio-proxy"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326032    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abf1311-b409-4ec0-bf10-39853bdb5156" containerName="payload-pre-processing"
Nov 03 21:51:42.326133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.326038    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea6de24-9086-4419-9046-1f221d5a7a8e" containerName="payload-pre-processing"
Nov 03 21:51:42.331122 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.331100    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.385832 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.385794    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"]
Nov 03 21:51:42.432619 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.432590    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:51:42.432619 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.432604    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:51:42.456947 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.456909    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.456947 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.456949    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fxf\" (UniqueName: \"kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.457235 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.457187    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.559067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.559025    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.559294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.559151    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.559294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.559179    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fxf\" (UniqueName: \"kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.559793 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.559767    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.561780 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.561756    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.616361 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.616233    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fxf\" (UniqueName: \"kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf\") pod \"maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:42.644728 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:42.644695    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:43.022406 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:43.022373    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"]
Nov 03 21:51:43.024613 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:51:43.024580    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3765aaf1_c914_4660_bd6f_8d594d48c560.slice/crio-d6abbb9867b4dc40df55515953f9318896aadc77f80d8ac074476e452db0a345 WatchSource:0}: Error finding container d6abbb9867b4dc40df55515953f9318896aadc77f80d8ac074476e452db0a345: Status 404 returned error can't find the container with id d6abbb9867b4dc40df55515953f9318896aadc77f80d8ac074476e452db0a345
Nov 03 21:51:43.207050 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:43.207012    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n" event={"ID":"3765aaf1-c914-4660-bd6f-8d594d48c560","Type":"ContainerStarted","Data":"d6abbb9867b4dc40df55515953f9318896aadc77f80d8ac074476e452db0a345"}
Nov 03 21:51:43.937057 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:43.937022    5244 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3765aaf1_c914_4660_bd6f_8d594d48c560.slice/crio-b780e0970bd7dbe73036c26df320ea647661c8a0551e1e3f46dc1fc2a1b4c1fe.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 21:51:44.213404 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:44.213278    5244 generic.go:356] "Generic (PLEG): container finished" podID="3765aaf1-c914-4660-bd6f-8d594d48c560" containerID="b780e0970bd7dbe73036c26df320ea647661c8a0551e1e3f46dc1fc2a1b4c1fe" exitCode=0
Nov 03 21:51:44.213404 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:44.213371    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n" event={"ID":"3765aaf1-c914-4660-bd6f-8d594d48c560","Type":"ContainerDied","Data":"b780e0970bd7dbe73036c26df320ea647661c8a0551e1e3f46dc1fc2a1b4c1fe"}
Nov 03 21:51:45.498766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.498737    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:45.613067 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.612969    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca\") pod \"3765aaf1-c914-4660-bd6f-8d594d48c560\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") "
Nov 03 21:51:45.613859 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.613835    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca\") pod \"3765aaf1-c914-4660-bd6f-8d594d48c560\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") "
Nov 03 21:51:45.614453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.614433    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf\" (UniqueName: \"kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf\") pod \"3765aaf1-c914-4660-bd6f-8d594d48c560\" (UID: \"3765aaf1-c914-4660-bd6f-8d594d48c560\") "
Nov 03 21:51:45.614944 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.613653    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca" pod "3765aaf1-c914-4660-bd6f-8d594d48c560" (UID: "3765aaf1-c914-4660-bd6f-8d594d48c560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:51:45.615478 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.615455    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:45.621619 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.620921    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf" pod "3765aaf1-c914-4660-bd6f-8d594d48c560" (UID: "3765aaf1-c914-4660-bd6f-8d594d48c560"). InnerVolumeSpecName "kube-api-access-q6fxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:51:45.621827 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.621803    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca" pod "3765aaf1-c914-4660-bd6f-8d594d48c560" (UID: "3765aaf1-c914-4660-bd6f-8d594d48c560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:51:45.721125 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.721077    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3765aaf1-c914-4660-bd6f-8d594d48c560-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:45.721125 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:45.721111    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6fxf\" (UniqueName: \"kubernetes.io/projected/3765aaf1-c914-4660-bd6f-8d594d48c560-kube-api-access-q6fxf\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:51:46.306182 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:46.306149    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"
Nov 03 21:51:46.306415 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:46.306154    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n" event={"ID":"3765aaf1-c914-4660-bd6f-8d594d48c560","Type":"ContainerDied","Data":"d6abbb9867b4dc40df55515953f9318896aadc77f80d8ac074476e452db0a345"}
Nov 03 21:51:46.306415 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:51:46.306262    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6abbb9867b4dc40df55515953f9318896aadc77f80d8ac074476e452db0a345"
Nov 03 21:51:52.436641 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:51:52.436604    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:52:22.437107 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:52:22.437063    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:52:24.027823 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:24.027786    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:52:32.065553 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.065514    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"]
Nov 03 21:52:32.065988 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.065974    5244 cpu_manager.go:414] "containerMap: removing container" podUID="3765aaf1-c914-4660-bd6f-8d594d48c560" containerName="revoke-keys"
Nov 03 21:52:32.066031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.065988    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3765aaf1-c914-4660-bd6f-8d594d48c560" containerName="revoke-keys"
Nov 03 21:52:32.066071 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.066057    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="3765aaf1-c914-4660-bd6f-8d594d48c560" containerName="revoke-keys"
Nov 03 21:52:32.070584 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.070567    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.172916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.172882    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ae2f6f\""
Nov 03 21:52:32.172916 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.172894    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ae2f6f-dockercfg-5lpr9\""
Nov 03 21:52:32.175115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.175094    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"]
Nov 03 21:52:32.204100 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.204068    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rfz\" (UniqueName: \"kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.204273 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.204146    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.305003 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.304961    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.305190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.305081    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rfz\" (UniqueName: \"kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.305774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.305753    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.361911 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.361823    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rfz\" (UniqueName: \"kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz\") pod \"payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.383853 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.383815    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:32.762551 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.762349    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"]
Nov 03 21:52:32.765139 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:52:32.765104    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3da146_c661_4b1f_a396_380b2687e39f.slice/crio-389e6f22eb7955eb1d6138a57ed301d42fccf1f489ed5327f4968ef7eb93a8e3 WatchSource:0}: Error finding container 389e6f22eb7955eb1d6138a57ed301d42fccf1f489ed5327f4968ef7eb93a8e3: Status 404 returned error can't find the container with id 389e6f22eb7955eb1d6138a57ed301d42fccf1f489ed5327f4968ef7eb93a8e3
Nov 03 21:52:32.872470 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.872438    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"]
Nov 03 21:52:32.875967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.875942    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:32.933536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:32.933494    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"]
Nov 03 21:52:33.012285 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.012194    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.012285 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.012238    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cglt6\" (UniqueName: \"kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6\") pod \"payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.113246 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.113203    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.113246 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.113251    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglt6\" (UniqueName: \"kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6\") pod \"payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.113925 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.113906    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.169766 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.169736    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglt6\" (UniqueName: \"kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6\") pod \"payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.189219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.189188    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:33.564171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.564143    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"]
Nov 03 21:52:33.566874 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:52:33.566842    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d261c19_3973_47a0_8b5c_6d11af2558b7.slice/crio-a507b2bd4430bb04bf513bc802a2b9bd88c3096e938a62094b0b5afe12c5e1ce WatchSource:0}: Error finding container a507b2bd4430bb04bf513bc802a2b9bd88c3096e938a62094b0b5afe12c5e1ce: Status 404 returned error can't find the container with id a507b2bd4430bb04bf513bc802a2b9bd88c3096e938a62094b0b5afe12c5e1ce
Nov 03 21:52:33.634351 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.630520    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw" event={"ID":"0d261c19-3973-47a0-8b5c-6d11af2558b7","Type":"ContainerStarted","Data":"a507b2bd4430bb04bf513bc802a2b9bd88c3096e938a62094b0b5afe12c5e1ce"}
Nov 03 21:52:33.638438 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.637799    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq" event={"ID":"7a3da146-c661-4b1f-a396-380b2687e39f","Type":"ContainerStarted","Data":"557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429"}
Nov 03 21:52:33.638438 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.637867    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq" event={"ID":"7a3da146-c661-4b1f-a396-380b2687e39f","Type":"ContainerStarted","Data":"389e6f22eb7955eb1d6138a57ed301d42fccf1f489ed5327f4968ef7eb93a8e3"}
Nov 03 21:52:33.638438 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.637925    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:33.751922 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:33.751870    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq" podStartSLOduration=1.751856165 podStartE2EDuration="1.751856165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:33.751120736 +0000 UTC m=+2224.735117316" watchObservedRunningTime="2026-11-03 21:52:33.751856165 +0000 UTC m=+2224.735852764"
Nov 03 21:52:34.646075 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:34.646038    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw" event={"ID":"0d261c19-3973-47a0-8b5c-6d11af2558b7","Type":"ContainerStarted","Data":"0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e"}
Nov 03 21:52:34.646641 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:34.646247    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:34.760198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:34.760138    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw" podStartSLOduration=2.760122406 podStartE2EDuration="2.760122406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:52:34.758708892 +0000 UTC m=+2225.742705474" watchObservedRunningTime="2026-11-03 21:52:34.760122406 +0000 UTC m=+2225.744119017"
Nov 03 21:52:44.648272 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:44.648235    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:52:45.653515 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:45.653487    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:52:52.437846 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:52:52.437804    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:52:54.044994 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:54.044938    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"]
Nov 03 21:52:54.094688 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:54.094653    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-62cf5c-r879n"]
Nov 03 21:52:55.136447 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:55.136409    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3765aaf1-c914-4660-bd6f-8d594d48c560" path="/var/lib/kubelet/pods/3765aaf1-c914-4660-bd6f-8d594d48c560/volumes"
Nov 03 21:52:58.418322 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:52:58.418293    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:53:04.397046 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:04.397013    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"]
Nov 03 21:53:04.397624 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:04.397267    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" containerID="cri-o://0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7" gracePeriod=30
Nov 03 21:53:06.920771 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:06.920685    5244 patch_prober.go:29] interesting pod/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:53:06.921143 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:06.920762    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:53:07.684632 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:07.684595    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6_d99f73bc-8f0e-4778-bed8-cfae7ed9dce1/payload-processing/0.log"
Nov 03 21:53:09.689715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.689686    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:53:09.793259 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793224    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793266    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793284    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793309    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793355    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793449 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793422    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793498    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793550    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.793723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793604    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:53:09.793723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793639    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:53:09.793723 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793680    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert\") pod \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\" (UID: \"f0f75301-9554-4ea7-b78f-2bab94f03ef7\") "
Nov 03 21:53:09.794000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793776    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:53:09.794000 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.793787    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:53:09.794275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.794112    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.794275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.794133    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.794275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.794147    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.794275 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.794160    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.794602 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.794567    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:09.796031 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.795987    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:53:09.796146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.796051    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:53:09.796351 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.796301    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:09.796477 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.796374    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl" pod "f0f75301-9554-4ea7-b78f-2bab94f03ef7" (UID: "f0f75301-9554-4ea7-b78f-2bab94f03ef7"). InnerVolumeSpecName "kube-api-access-w6hbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:09.870260 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.870225    5244 generic.go:356] "Generic (PLEG): container finished" podID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerID="0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7" exitCode=0
Nov 03 21:53:09.870482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.870286    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"
Nov 03 21:53:09.870482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.870307    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" event={"ID":"f0f75301-9554-4ea7-b78f-2bab94f03ef7","Type":"ContainerDied","Data":"0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7"}
Nov 03 21:53:09.870482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.870410    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx" event={"ID":"f0f75301-9554-4ea7-b78f-2bab94f03ef7","Type":"ContainerDied","Data":"81947f0e1cb9d311ac3b713cf0977be8f4de7d5a1abb77c9e00b4b20497531f5"}
Nov 03 21:53:09.870482 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.870441    5244 scope.go:122] "RemoveContainer" containerID="0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7"
Nov 03 21:53:09.881991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.881968    5244 scope.go:122] "RemoveContainer" containerID="0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7"
Nov 03 21:53:09.882297 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:53:09.882277    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7\": container with ID starting with 0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7 not found: ID does not exist" containerID="0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7"
Nov 03 21:53:09.882431 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.882305    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7"} err="failed to get container status \"0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7\": rpc error: code = NotFound desc = could not find container \"0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7\": container with ID starting with 0fb69d69756a6196008dc56b5199081806ce82c6f471781802c5b2e0db8a00f7 not found: ID does not exist"
Nov 03 21:53:09.895262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.895223    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.895262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.895259    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.895434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.895275    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6hbl\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-kube-api-access-w6hbl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.895434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.895293    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:09.895434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:09.895307    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f75301-9554-4ea7-b78f-2bab94f03ef7-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:10.016503 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.016469    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"]
Nov 03 21:53:10.020432 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.020405    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d71a74-openshift-default-5894658649-d95zx"]
Nov 03 21:53:10.742252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.738496    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"]
Nov 03 21:53:10.742252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.739649    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy"
Nov 03 21:53:10.742252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.739669    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy"
Nov 03 21:53:10.742252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.739856    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" containerName="istio-proxy"
Nov 03 21:53:10.747319 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.747291    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:10.795886 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.795845    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"]
Nov 03 21:53:10.799263 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.799237    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:53:10.799263 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.799255    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:53:10.910448 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.910419    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:10.910634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.910471    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:10.910695 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:10.910669    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jtn\" (UniqueName: \"kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.011928 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.011827    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jtn\" (UniqueName: \"kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.012104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.011940    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.012104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.011972    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.012601 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.012579    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.014660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.014634    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.097925 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.089792    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jtn\" (UniqueName: \"kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn\") pod \"maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.139869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.139828    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f75301-9554-4ea7-b78f-2bab94f03ef7" path="/var/lib/kubelet/pods/f0f75301-9554-4ea7-b78f-2bab94f03ef7/volumes"
Nov 03 21:53:11.360039 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.359949    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:11.734540 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.734514    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"]
Nov 03 21:53:11.737137 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:53:11.737101    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059e07b1_ee7e_498e_8ba5_9a2a79152f31.slice/crio-8bcd95ca450d88db40e444199e296b4639a8804f7604039e26d0956764a086d1 WatchSource:0}: Error finding container 8bcd95ca450d88db40e444199e296b4639a8804f7604039e26d0956764a086d1: Status 404 returned error can't find the container with id 8bcd95ca450d88db40e444199e296b4639a8804f7604039e26d0956764a086d1
Nov 03 21:53:11.890923 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:11.890885    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5" event={"ID":"059e07b1-ee7e-498e-8ba5-9a2a79152f31","Type":"ContainerStarted","Data":"8bcd95ca450d88db40e444199e296b4639a8804f7604039e26d0956764a086d1"}
Nov 03 21:53:12.899261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:12.899165    5244 generic.go:356] "Generic (PLEG): container finished" podID="059e07b1-ee7e-498e-8ba5-9a2a79152f31" containerID="cee7e5112b01a2ecd0c1d52162fcf23b490ab9afd9d6e907016aeb69b92fa559" exitCode=0
Nov 03 21:53:12.899744 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:12.899266    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5" event={"ID":"059e07b1-ee7e-498e-8ba5-9a2a79152f31","Type":"ContainerDied","Data":"cee7e5112b01a2ecd0c1d52162fcf23b490ab9afd9d6e907016aeb69b92fa559"}
Nov 03 21:53:14.066578 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.066548    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:14.143670 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.143630    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca\") pod \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") "
Nov 03 21:53:14.143851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.143709    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca\") pod \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") "
Nov 03 21:53:14.143851 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.143809    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn\" (UniqueName: \"kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn\") pod \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\" (UID: \"059e07b1-ee7e-498e-8ba5-9a2a79152f31\") "
Nov 03 21:53:14.144014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.143996    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca" pod "059e07b1-ee7e-498e-8ba5-9a2a79152f31" (UID: "059e07b1-ee7e-498e-8ba5-9a2a79152f31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:14.144352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.144234    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:14.146148 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.146126    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn" pod "059e07b1-ee7e-498e-8ba5-9a2a79152f31" (UID: "059e07b1-ee7e-498e-8ba5-9a2a79152f31"). InnerVolumeSpecName "kube-api-access-w6jtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:14.146238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.146162    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca" pod "059e07b1-ee7e-498e-8ba5-9a2a79152f31" (UID: "059e07b1-ee7e-498e-8ba5-9a2a79152f31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:53:14.245521 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.245488    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059e07b1-ee7e-498e-8ba5-9a2a79152f31-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:14.245521 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.245520    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6jtn\" (UniqueName: \"kubernetes.io/projected/059e07b1-ee7e-498e-8ba5-9a2a79152f31-kube-api-access-w6jtn\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:14.915047 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.915013    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5" event={"ID":"059e07b1-ee7e-498e-8ba5-9a2a79152f31","Type":"ContainerDied","Data":"8bcd95ca450d88db40e444199e296b4639a8804f7604039e26d0956764a086d1"}
Nov 03 21:53:14.915047 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.915042    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"
Nov 03 21:53:14.915276 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:14.915067    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bcd95ca450d88db40e444199e296b4639a8804f7604039e26d0956764a086d1"
Nov 03 21:53:20.428802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.428757    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"]
Nov 03 21:53:20.429293 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.429046    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4" podUID="cd73fece-1e0e-4244-ab0f-1f9b14786472" containerName="payload-processing" containerID="cri-o://7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030" gracePeriod=30
Nov 03 21:53:20.488459 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.488418    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"]
Nov 03 21:53:20.488696 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.488671    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j" podUID="137991db-2ac5-4ca6-8e0b-7701035514ba" containerName="payload-pre-processing" containerID="cri-o://4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4" gracePeriod=30
Nov 03 21:53:20.776451 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.776426    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:53:20.780530 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.780506    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:53:20.830366 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.830305    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume\") pod \"cd73fece-1e0e-4244-ab0f-1f9b14786472\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") "
Nov 03 21:53:20.830577 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.830421    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume\") pod \"137991db-2ac5-4ca6-8e0b-7701035514ba\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") "
Nov 03 21:53:20.830577 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.830453    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6\" (UniqueName: \"kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6\") pod \"cd73fece-1e0e-4244-ab0f-1f9b14786472\" (UID: \"cd73fece-1e0e-4244-ab0f-1f9b14786472\") "
Nov 03 21:53:20.830786 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.830758    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume" pod "cd73fece-1e0e-4244-ab0f-1f9b14786472" (UID: "cd73fece-1e0e-4244-ab0f-1f9b14786472"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:20.830862 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.830811    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume" pod "137991db-2ac5-4ca6-8e0b-7701035514ba" (UID: "137991db-2ac5-4ca6-8e0b-7701035514ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:20.830921 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.830896    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd73fece-1e0e-4244-ab0f-1f9b14786472-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:20.832808 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.832780    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6" pod "cd73fece-1e0e-4244-ab0f-1f9b14786472" (UID: "cd73fece-1e0e-4244-ab0f-1f9b14786472"). InnerVolumeSpecName "kube-api-access-s5wd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:20.935635 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.931452    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7\" (UniqueName: \"kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7\") pod \"137991db-2ac5-4ca6-8e0b-7701035514ba\" (UID: \"137991db-2ac5-4ca6-8e0b-7701035514ba\") "
Nov 03 21:53:20.935635 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.932088    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137991db-2ac5-4ca6-8e0b-7701035514ba-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:20.935635 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.932107    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wd6\" (UniqueName: \"kubernetes.io/projected/cd73fece-1e0e-4244-ab0f-1f9b14786472-kube-api-access-s5wd6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:20.935635 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.934789    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7" pod "137991db-2ac5-4ca6-8e0b-7701035514ba" (UID: "137991db-2ac5-4ca6-8e0b-7701035514ba"). InnerVolumeSpecName "kube-api-access-f55k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.968464    5244 generic.go:356] "Generic (PLEG): container finished" podID="137991db-2ac5-4ca6-8e0b-7701035514ba" containerID="4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4" exitCode=0
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.968527    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.968637    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j" event={"ID":"137991db-2ac5-4ca6-8e0b-7701035514ba","Type":"ContainerDied","Data":"4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4"}
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.968687    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j" event={"ID":"137991db-2ac5-4ca6-8e0b-7701035514ba","Type":"ContainerDied","Data":"31cecda5888fdf7d7bfb8772981be3dc658c13f3fc15d45ebba58c0df8367246"}
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.968718    5244 scope.go:122] "RemoveContainer" containerID="4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4"
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.974832    5244 generic.go:356] "Generic (PLEG): container finished" podID="cd73fece-1e0e-4244-ab0f-1f9b14786472" containerID="7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030" exitCode=0
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.974957    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4" event={"ID":"cd73fece-1e0e-4244-ab0f-1f9b14786472","Type":"ContainerDied","Data":"7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030"}
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.975005    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4" event={"ID":"cd73fece-1e0e-4244-ab0f-1f9b14786472","Type":"ContainerDied","Data":"de8933bf54fd2e303815ed8571e31699e89c9506e773189fefc093ccd5449b0c"}
Nov 03 21:53:20.976353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:20.975085    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"
Nov 03 21:53:21.016727 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.016444    5244 scope.go:122] "RemoveContainer" containerID="4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4"
Nov 03 21:53:21.016907 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:53:21.016878    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4\": container with ID starting with 4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4 not found: ID does not exist" containerID="4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4"
Nov 03 21:53:21.017025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.016919    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4"} err="failed to get container status \"4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4\": rpc error: code = NotFound desc = could not find container \"4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4\": container with ID starting with 4cb96eeb59018df8f18470c0b5a9078aade6480204a0da3a9a9cd123f83012a4 not found: ID does not exist"
Nov 03 21:53:21.017025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.016950    5244 scope.go:122] "RemoveContainer" containerID="7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030"
Nov 03 21:53:21.029076 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.029038    5244 scope.go:122] "RemoveContainer" containerID="7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030"
Nov 03 21:53:21.029501 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:53:21.029468    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030\": container with ID starting with 7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030 not found: ID does not exist" containerID="7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030"
Nov 03 21:53:21.029634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.029510    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030"} err="failed to get container status \"7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030\": rpc error: code = NotFound desc = could not find container \"7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030\": container with ID starting with 7c583bfb6a10c7f1a83648013339c79b44a3c1c114eb294e480a1df594772030 not found: ID does not exist"
Nov 03 21:53:21.033179 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.033149    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55k7\" (UniqueName: \"kubernetes.io/projected/137991db-2ac5-4ca6-8e0b-7701035514ba-kube-api-access-f55k7\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:21.122830 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.122794    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"]
Nov 03 21:53:21.125091 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.125065    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3940f8-6d464766bb-xgch4"]
Nov 03 21:53:21.135672 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.135650    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd73fece-1e0e-4244-ab0f-1f9b14786472" path="/var/lib/kubelet/pods/cd73fece-1e0e-4244-ab0f-1f9b14786472/volumes"
Nov 03 21:53:21.232962 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.232926    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"]
Nov 03 21:53:21.236638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:21.236611    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3940f8-8f5c875ff-tz79j"]
Nov 03 21:53:22.440315 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:53:22.440280    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:53:23.136261 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:23.136224    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137991db-2ac5-4ca6-8e0b-7701035514ba" path="/var/lib/kubelet/pods/137991db-2ac5-4ca6-8e0b-7701035514ba/volumes"
Nov 03 21:53:29.999640 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:29.999605    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"]
Nov 03 21:53:30.000115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000030    5244 cpu_manager.go:414] "containerMap: removing container" podUID="059e07b1-ee7e-498e-8ba5-9a2a79152f31" containerName="revoke-keys"
Nov 03 21:53:30.000115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000041    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059e07b1-ee7e-498e-8ba5-9a2a79152f31" containerName="revoke-keys"
Nov 03 21:53:30.000115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000066    5244 cpu_manager.go:414] "containerMap: removing container" podUID="cd73fece-1e0e-4244-ab0f-1f9b14786472" containerName="payload-processing"
Nov 03 21:53:30.000115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000072    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd73fece-1e0e-4244-ab0f-1f9b14786472" containerName="payload-processing"
Nov 03 21:53:30.000115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000085    5244 cpu_manager.go:414] "containerMap: removing container" podUID="137991db-2ac5-4ca6-8e0b-7701035514ba" containerName="payload-pre-processing"
Nov 03 21:53:30.000115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000091    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137991db-2ac5-4ca6-8e0b-7701035514ba" containerName="payload-pre-processing"
Nov 03 21:53:30.000318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000158    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="059e07b1-ee7e-498e-8ba5-9a2a79152f31" containerName="revoke-keys"
Nov 03 21:53:30.000318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000170    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd73fece-1e0e-4244-ab0f-1f9b14786472" containerName="payload-processing"
Nov 03 21:53:30.000318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.000176    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="137991db-2ac5-4ca6-8e0b-7701035514ba" containerName="payload-pre-processing"
Nov 03 21:53:30.005079 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.005058    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.107177 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.107142    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:53:30.107177 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.107171    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:53:30.109149 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.109128    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"]
Nov 03 21:53:30.138719 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.138686    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.138870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.138730    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdkv\" (UniqueName: \"kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.138870 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.138754    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.239850 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.239804    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.240085 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.240049    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdkv\" (UniqueName: \"kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.240165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.240115    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.240516 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.240494    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.242563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.242546    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.297158 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.297078    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdkv\" (UniqueName: \"kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv\") pod \"maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.318057 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.318019    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:30.692295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:30.692264    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"]
Nov 03 21:53:30.694505 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:53:30.694472    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5817620_36f9_4262_9da1_0285067d5ced.slice/crio-b579281e2283789b7a7f06b6a553dd44ef6499ed2828b11225dc07476ef297af WatchSource:0}: Error finding container b579281e2283789b7a7f06b6a553dd44ef6499ed2828b11225dc07476ef297af: Status 404 returned error can't find the container with id b579281e2283789b7a7f06b6a553dd44ef6499ed2828b11225dc07476ef297af
Nov 03 21:53:31.036419 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:31.036377    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n" event={"ID":"f5817620-36f9-4262-9da1-0285067d5ced","Type":"ContainerStarted","Data":"fa472909057bcb82a5632189679c74204179a2f52b41b55822045d0d29dd1a89"}
Nov 03 21:53:31.036874 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:31.036437    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n" event={"ID":"f5817620-36f9-4262-9da1-0285067d5ced","Type":"ContainerStarted","Data":"b579281e2283789b7a7f06b6a553dd44ef6499ed2828b11225dc07476ef297af"}
Nov 03 21:53:31.157637 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:31.157585    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n" podStartSLOduration=2.157568542 podStartE2EDuration="2.157568542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:53:31.156545948 +0000 UTC m=+2282.140542506" watchObservedRunningTime="2026-11-03 21:53:31.157568542 +0000 UTC m=+2282.141565102"
Nov 03 21:53:32.046103 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:32.046071    5244 generic.go:356] "Generic (PLEG): container finished" podID="f5817620-36f9-4262-9da1-0285067d5ced" containerID="fa472909057bcb82a5632189679c74204179a2f52b41b55822045d0d29dd1a89" exitCode=0
Nov 03 21:53:32.046569 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:32.046153    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n" event={"ID":"f5817620-36f9-4262-9da1-0285067d5ced","Type":"ContainerDied","Data":"fa472909057bcb82a5632189679c74204179a2f52b41b55822045d0d29dd1a89"}
Nov 03 21:53:33.198008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.197984    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:33.276952 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.276912    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca\") pod \"f5817620-36f9-4262-9da1-0285067d5ced\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") "
Nov 03 21:53:33.277152 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.276989    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca\") pod \"f5817620-36f9-4262-9da1-0285067d5ced\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") "
Nov 03 21:53:33.277152 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.277040    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv\" (UniqueName: \"kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv\") pod \"f5817620-36f9-4262-9da1-0285067d5ced\" (UID: \"f5817620-36f9-4262-9da1-0285067d5ced\") "
Nov 03 21:53:33.277299 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.277272    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca" pod "f5817620-36f9-4262-9da1-0285067d5ced" (UID: "f5817620-36f9-4262-9da1-0285067d5ced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:53:33.277467 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.277436    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:33.279231 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.279203    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca" pod "f5817620-36f9-4262-9da1-0285067d5ced" (UID: "f5817620-36f9-4262-9da1-0285067d5ced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:53:33.279392 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.279290    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv" pod "f5817620-36f9-4262-9da1-0285067d5ced" (UID: "f5817620-36f9-4262-9da1-0285067d5ced"). InnerVolumeSpecName "kube-api-access-srdkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:53:33.378272 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.378183    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5817620-36f9-4262-9da1-0285067d5ced-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:33.378272 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:33.378212    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdkv\" (UniqueName: \"kubernetes.io/projected/f5817620-36f9-4262-9da1-0285067d5ced-kube-api-access-srdkv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:53:34.059895 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:34.059858    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n" event={"ID":"f5817620-36f9-4262-9da1-0285067d5ced","Type":"ContainerDied","Data":"b579281e2283789b7a7f06b6a553dd44ef6499ed2828b11225dc07476ef297af"}
Nov 03 21:53:34.059895 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:34.059898    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"
Nov 03 21:53:34.060128 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:34.059918    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b579281e2283789b7a7f06b6a553dd44ef6499ed2828b11225dc07476ef297af"
Nov 03 21:53:46.762074 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:53:46.762032    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:53:52.439205 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:53:52.439164    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:54:22.438158 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:54:22.438112    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:54:24.784192 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.784155    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"]
Nov 03 21:54:24.784661 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.784647    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f5817620-36f9-4262-9da1-0285067d5ced" containerName="revoke-keys"
Nov 03 21:54:24.784708 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.784663    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5817620-36f9-4262-9da1-0285067d5ced" containerName="revoke-keys"
Nov 03 21:54:24.784758 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.784748    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5817620-36f9-4262-9da1-0285067d5ced" containerName="revoke-keys"
Nov 03 21:54:24.787869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.787854    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.845365 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.845312    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:54:24.881598 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.881556    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.881974 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.881949    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cwv\" (UniqueName: \"kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.882121 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.882106    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.896600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.896562    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:54:24.902562 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.902511    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"]
Nov 03 21:54:24.983874 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.983831    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.984071 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.983889    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cwv\" (UniqueName: \"kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.984071 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.983934    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.984575 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.984552    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:24.986509 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:24.986488    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:25.076456 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:25.076382    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cwv\" (UniqueName: \"kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv\") pod \"maas-api-revoke-sub-fd1c0a7c-q4vs8\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:25.101387 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:25.101350    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:25.474265 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:25.474238    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"]
Nov 03 21:54:25.477104 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:54:25.477070    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed61697_2be1_4c26_8cf2_9e20535dab3e.slice/crio-0740191f4b7ae79cffb197e824df6269bbd2d62ff3ba607e6fd0eef9e3c3deb1 WatchSource:0}: Error finding container 0740191f4b7ae79cffb197e824df6269bbd2d62ff3ba607e6fd0eef9e3c3deb1: Status 404 returned error can't find the container with id 0740191f4b7ae79cffb197e824df6269bbd2d62ff3ba607e6fd0eef9e3c3deb1
Nov 03 21:54:26.350657 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:26.350628    5244 generic.go:356] "Generic (PLEG): container finished" podID="1ed61697-2be1-4c26-8cf2-9e20535dab3e" containerID="b8f16e35569f9180b9124c5d008d6c03ad355a9adff09e088e7d30919fb49495" exitCode=0
Nov 03 21:54:26.351014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:26.350717    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8" event={"ID":"1ed61697-2be1-4c26-8cf2-9e20535dab3e","Type":"ContainerDied","Data":"b8f16e35569f9180b9124c5d008d6c03ad355a9adff09e088e7d30919fb49495"}
Nov 03 21:54:26.351014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:26.350777    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8" event={"ID":"1ed61697-2be1-4c26-8cf2-9e20535dab3e","Type":"ContainerStarted","Data":"0740191f4b7ae79cffb197e824df6269bbd2d62ff3ba607e6fd0eef9e3c3deb1"}
Nov 03 21:54:27.501609 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.501582    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:27.612822 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.612726    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv\" (UniqueName: \"kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv\") pod \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") "
Nov 03 21:54:27.612995 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.612864    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca\") pod \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") "
Nov 03 21:54:27.613041 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.612993    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca\") pod \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\" (UID: \"1ed61697-2be1-4c26-8cf2-9e20535dab3e\") "
Nov 03 21:54:27.613360 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.613311    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca" pod "1ed61697-2be1-4c26-8cf2-9e20535dab3e" (UID: "1ed61697-2be1-4c26-8cf2-9e20535dab3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:27.615357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.615295    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca" pod "1ed61697-2be1-4c26-8cf2-9e20535dab3e" (UID: "1ed61697-2be1-4c26-8cf2-9e20535dab3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:54:27.615496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.615470    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv" pod "1ed61697-2be1-4c26-8cf2-9e20535dab3e" (UID: "1ed61697-2be1-4c26-8cf2-9e20535dab3e"). InnerVolumeSpecName "kube-api-access-94cwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:27.714985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.714946    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:27.714985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.714980    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed61697-2be1-4c26-8cf2-9e20535dab3e-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:27.714985 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:27.714992    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94cwv\" (UniqueName: \"kubernetes.io/projected/1ed61697-2be1-4c26-8cf2-9e20535dab3e-kube-api-access-94cwv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:28.362634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:28.362602    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"
Nov 03 21:54:28.362634 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:28.362618    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8" event={"ID":"1ed61697-2be1-4c26-8cf2-9e20535dab3e","Type":"ContainerDied","Data":"0740191f4b7ae79cffb197e824df6269bbd2d62ff3ba607e6fd0eef9e3c3deb1"}
Nov 03 21:54:28.362855 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:28.362668    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0740191f4b7ae79cffb197e824df6269bbd2d62ff3ba607e6fd0eef9e3c3deb1"
Nov 03 21:54:29.679965 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:29.679928    5244 scope.go:122] "RemoveContainer" containerID="5f39a9681fafbb5f1dfaea987d753830b8f24c138ecc8c27e61652610a870f4b"
Nov 03 21:54:36.819942 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:36.819849    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"]
Nov 03 21:54:36.820405 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:36.820148    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw" podUID="0d261c19-3973-47a0-8b5c-6d11af2558b7" containerName="payload-processing" containerID="cri-o://0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e" gracePeriod=30
Nov 03 21:54:36.870880 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:36.870847    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"]
Nov 03 21:54:36.871082 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:36.871048    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq" podUID="7a3da146-c661-4b1f-a396-380b2687e39f" containerName="payload-pre-processing" containerID="cri-o://557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429" gracePeriod=30
Nov 03 21:54:37.123826 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.123797    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:54:37.158839 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.158815    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:54:37.224570 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.224540    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6\" (UniqueName: \"kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6\") pod \"0d261c19-3973-47a0-8b5c-6d11af2558b7\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") "
Nov 03 21:54:37.224747 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.224609    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume\") pod \"0d261c19-3973-47a0-8b5c-6d11af2558b7\" (UID: \"0d261c19-3973-47a0-8b5c-6d11af2558b7\") "
Nov 03 21:54:37.225035 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.224994    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume" pod "0d261c19-3973-47a0-8b5c-6d11af2558b7" (UID: "0d261c19-3973-47a0-8b5c-6d11af2558b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:37.226932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.226904    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6" pod "0d261c19-3973-47a0-8b5c-6d11af2558b7" (UID: "0d261c19-3973-47a0-8b5c-6d11af2558b7"). InnerVolumeSpecName "kube-api-access-cglt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:37.342390 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.327212    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume\") pod \"7a3da146-c661-4b1f-a396-380b2687e39f\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") "
Nov 03 21:54:37.342390 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.327300    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz\" (UniqueName: \"kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz\") pod \"7a3da146-c661-4b1f-a396-380b2687e39f\" (UID: \"7a3da146-c661-4b1f-a396-380b2687e39f\") "
Nov 03 21:54:37.342390 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.328016    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cglt6\" (UniqueName: \"kubernetes.io/projected/0d261c19-3973-47a0-8b5c-6d11af2558b7-kube-api-access-cglt6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:37.342390 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.328045    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d261c19-3973-47a0-8b5c-6d11af2558b7-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:37.342390 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.330703    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume" pod "7a3da146-c661-4b1f-a396-380b2687e39f" (UID: "7a3da146-c661-4b1f-a396-380b2687e39f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:37.346379 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.345269    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz" pod "7a3da146-c661-4b1f-a396-380b2687e39f" (UID: "7a3da146-c661-4b1f-a396-380b2687e39f"). InnerVolumeSpecName "kube-api-access-t2rfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:37.416920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.416834    5244 generic.go:356] "Generic (PLEG): container finished" podID="7a3da146-c661-4b1f-a396-380b2687e39f" containerID="557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429" exitCode=0
Nov 03 21:54:37.416920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.416879    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"
Nov 03 21:54:37.416920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.416904    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq" event={"ID":"7a3da146-c661-4b1f-a396-380b2687e39f","Type":"ContainerDied","Data":"557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429"}
Nov 03 21:54:37.417222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.416961    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq" event={"ID":"7a3da146-c661-4b1f-a396-380b2687e39f","Type":"ContainerDied","Data":"389e6f22eb7955eb1d6138a57ed301d42fccf1f489ed5327f4968ef7eb93a8e3"}
Nov 03 21:54:37.417222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.416985    5244 scope.go:122] "RemoveContainer" containerID="557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429"
Nov 03 21:54:37.418769 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.418742    5244 generic.go:356] "Generic (PLEG): container finished" podID="0d261c19-3973-47a0-8b5c-6d11af2558b7" containerID="0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e" exitCode=0
Nov 03 21:54:37.418945 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.418788    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw" event={"ID":"0d261c19-3973-47a0-8b5c-6d11af2558b7","Type":"ContainerDied","Data":"0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e"}
Nov 03 21:54:37.418945 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.418806    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"
Nov 03 21:54:37.418945 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.418834    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw" event={"ID":"0d261c19-3973-47a0-8b5c-6d11af2558b7","Type":"ContainerDied","Data":"a507b2bd4430bb04bf513bc802a2b9bd88c3096e938a62094b0b5afe12c5e1ce"}
Nov 03 21:54:37.429266 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.429240    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a3da146-c661-4b1f-a396-380b2687e39f-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:37.429394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.429272    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2rfz\" (UniqueName: \"kubernetes.io/projected/7a3da146-c661-4b1f-a396-380b2687e39f-kube-api-access-t2rfz\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:37.429932 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.429909    5244 scope.go:122] "RemoveContainer" containerID="557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429"
Nov 03 21:54:37.430305 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:54:37.430237    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429\": container with ID starting with 557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429 not found: ID does not exist" containerID="557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429"
Nov 03 21:54:37.430305 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.430273    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429"} err="failed to get container status \"557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429\": rpc error: code = NotFound desc = could not find container \"557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429\": container with ID starting with 557a0634b1668aa8ab5207a1ee947a7a9d3f510a567a0a1f782ef60f48583429 not found: ID does not exist"
Nov 03 21:54:37.430305 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.430297    5244 scope.go:122] "RemoveContainer" containerID="0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e"
Nov 03 21:54:37.439119 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.439101    5244 scope.go:122] "RemoveContainer" containerID="0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e"
Nov 03 21:54:37.439436 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:54:37.439415    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e\": container with ID starting with 0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e not found: ID does not exist" containerID="0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e"
Nov 03 21:54:37.439499 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.439445    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e"} err="failed to get container status \"0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e\": rpc error: code = NotFound desc = could not find container \"0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e\": container with ID starting with 0214f181a88a9da8facfc00a3500ee052fa0baecdb15925e5041a1351dff905e not found: ID does not exist"
Nov 03 21:54:37.579357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.579308    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"]
Nov 03 21:54:37.584735 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.584706    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ae2f6f-59499d4b99-55kjq"]
Nov 03 21:54:37.692630 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.692602    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"]
Nov 03 21:54:37.695595 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:37.695568    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ae2f6f-5656d4478d-8r9tw"]
Nov 03 21:54:38.610131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:38.610097    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"]
Nov 03 21:54:38.612904 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:38.612882    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ae2f6f-phd5n"]
Nov 03 21:54:39.135400 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:39.135366    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d261c19-3973-47a0-8b5c-6d11af2558b7" path="/var/lib/kubelet/pods/0d261c19-3973-47a0-8b5c-6d11af2558b7/volumes"
Nov 03 21:54:39.135778 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:39.135764    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3da146-c661-4b1f-a396-380b2687e39f" path="/var/lib/kubelet/pods/7a3da146-c661-4b1f-a396-380b2687e39f/volumes"
Nov 03 21:54:39.136127 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:39.136115    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5817620-36f9-4262-9da1-0285067d5ced" path="/var/lib/kubelet/pods/f5817620-36f9-4262-9da1-0285067d5ced/volumes"
Nov 03 21:54:45.274499 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:45.274464    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"]
Nov 03 21:54:45.275012 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:45.274682    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy" containerID="cri-o://6b4340e1dc4c68b56971a0003ec6f612178afc9e84b437527d20dfcef0dd8e15" gracePeriod=30
Nov 03 21:54:50.505989 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.505957    5244 generic.go:356] "Generic (PLEG): container finished" podID="6292679d-0d43-426e-bf2f-9ce73a137393" containerID="6b4340e1dc4c68b56971a0003ec6f612178afc9e84b437527d20dfcef0dd8e15" exitCode=0
Nov 03 21:54:50.506453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.506057    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" event={"ID":"6292679d-0d43-426e-bf2f-9ce73a137393","Type":"ContainerDied","Data":"6b4340e1dc4c68b56971a0003ec6f612178afc9e84b437527d20dfcef0dd8e15"}
Nov 03 21:54:50.506453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.506153    5244 scope.go:122] "RemoveContainer" containerID="563e3e555e7836fa6af413486e723363bbf812096a2069f78ac979543f3df746"
Nov 03 21:54:50.569748 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.569621    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:54:50.680919 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.680887    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.681094 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.680946    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.681094 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681014    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.681094 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681044    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.681271 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681230    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.681394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681303    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:50.681394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681318    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.682097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681523    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.682097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681617    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.682097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681657    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs\") pod \"6292679d-0d43-426e-bf2f-9ce73a137393\" (UID: \"6292679d-0d43-426e-bf2f-9ce73a137393\") "
Nov 03 21:54:50.682097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681814    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:54:50.682097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.681934    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:50.682097 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.682065    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:50.682531 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.682364    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.682531 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.682377    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:50.682531 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.682386    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6292679d-0d43-426e-bf2f-9ce73a137393-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.682531 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.682405    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.682531 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.682420    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.683669 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.683620    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:54:50.683769 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.683719    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:50.683914 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.683884    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "kube-api-access-4qvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:54:50.683958 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.683924    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy" pod "6292679d-0d43-426e-bf2f-9ce73a137393" (UID: "6292679d-0d43-426e-bf2f-9ce73a137393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:54:50.783399 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.783364    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6292679d-0d43-426e-bf2f-9ce73a137393-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.783399 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.783395    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvjv\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-kube-api-access-4qvjv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.783399 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.783407    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6292679d-0d43-426e-bf2f-9ce73a137393-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.783631 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.783415    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:50.783631 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:50.783424    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6292679d-0d43-426e-bf2f-9ce73a137393-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:54:51.513099 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:51.513066    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"
Nov 03 21:54:51.513589 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:51.513108    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d" event={"ID":"6292679d-0d43-426e-bf2f-9ce73a137393","Type":"ContainerDied","Data":"58039ccb570f90312eb6136b7a65a7a8b40913fb3cb55daf9092ce794a663144"}
Nov 03 21:54:51.513589 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:51.513172    5244 scope.go:122] "RemoveContainer" containerID="6b4340e1dc4c68b56971a0003ec6f612178afc9e84b437527d20dfcef0dd8e15"
Nov 03 21:54:51.619814 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:51.619778    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:54:51.678717 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:51.678684    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"]
Nov 03 21:54:51.682303 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:51.682276    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ae2f6f-openshift-default-d7ffdcb9-nhq9d"]
Nov 03 21:54:52.459659 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:54:52.459610    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:54:53.135796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:54:53.135761    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" path="/var/lib/kubelet/pods/6292679d-0d43-426e-bf2f-9ce73a137393/volumes"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.071173    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"]
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075188    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7a3da146-c661-4b1f-a396-380b2687e39f" containerName="payload-pre-processing"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075215    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3da146-c661-4b1f-a396-380b2687e39f" containerName="payload-pre-processing"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075230    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075238    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075291    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1ed61697-2be1-4c26-8cf2-9e20535dab3e" containerName="revoke-keys"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075300    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed61697-2be1-4c26-8cf2-9e20535dab3e" containerName="revoke-keys"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075323    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0d261c19-3973-47a0-8b5c-6d11af2558b7" containerName="payload-processing"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075349    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d261c19-3973-47a0-8b5c-6d11af2558b7" containerName="payload-processing"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075485    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075497    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed61697-2be1-4c26-8cf2-9e20535dab3e" containerName="revoke-keys"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075509    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d261c19-3973-47a0-8b5c-6d11af2558b7" containerName="payload-processing"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.075520    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3da146-c661-4b1f-a396-380b2687e39f" containerName="payload-pre-processing"
Nov 03 21:55:00.082219 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.081581    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.141355 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.137573    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"]
Nov 03 21:55:00.184341 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.184289    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eb3520\""
Nov 03 21:55:00.184517 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.184319    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eb3520-dockercfg-gzq6r\""
Nov 03 21:55:00.202060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.202028    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.202233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.202181    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9td5\" (UniqueName: \"kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5\") pod \"payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.250038 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.250004    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"]
Nov 03 21:55:00.250453 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.250440    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy"
Nov 03 21:55:00.250500 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.250454    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy"
Nov 03 21:55:00.250535 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.250525    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6292679d-0d43-426e-bf2f-9ce73a137393" containerName="istio-proxy"
Nov 03 21:55:00.253656 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.253639    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.303654 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.303620    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.303831 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.303729    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9td5\" (UniqueName: \"kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5\") pod \"payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.304430 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.304408    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.311241 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.311216    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"]
Nov 03 21:55:00.363989 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.363917    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9td5\" (UniqueName: \"kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5\") pod \"payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.405146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.405111    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px2v8\" (UniqueName: \"kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8\") pod \"payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.405357 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.405152    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.422967 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.422937    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:00.506398 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.506364    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px2v8\" (UniqueName: \"kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8\") pod \"payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.506563 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.506407    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.507130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.507109    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.573105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.573060    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px2v8\" (UniqueName: \"kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8\") pod \"payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:00.807280 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.807227    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"]
Nov 03 21:55:00.809829 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:55:00.809798    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fda315c_f98f_4687_8c2e_e01f4f293891.slice/crio-0a2cf742aba6f57459acb0229bd8152f8b4c7df225667951e532df41a079a5b7 WatchSource:0}: Error finding container 0a2cf742aba6f57459acb0229bd8152f8b4c7df225667951e532df41a079a5b7: Status 404 returned error can't find the container with id 0a2cf742aba6f57459acb0229bd8152f8b4c7df225667951e532df41a079a5b7
Nov 03 21:55:00.866498 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:00.866473    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:01.243675 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.243646    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"]
Nov 03 21:55:01.245875 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:55:01.245841    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe290523_3644_4f1c_8870_63743b82d309.slice/crio-be071f00d9f20df04f1d024171e136ecd823efcf108785d63883a75b977dfe33 WatchSource:0}: Error finding container be071f00d9f20df04f1d024171e136ecd823efcf108785d63883a75b977dfe33: Status 404 returned error can't find the container with id be071f00d9f20df04f1d024171e136ecd823efcf108785d63883a75b977dfe33
Nov 03 21:55:01.578380 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.578247    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" event={"ID":"fe290523-3644-4f1c-8870-63743b82d309","Type":"ContainerStarted","Data":"55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1"}
Nov 03 21:55:01.578380 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.578313    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:01.578380 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.578354    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" event={"ID":"fe290523-3644-4f1c-8870-63743b82d309","Type":"ContainerStarted","Data":"be071f00d9f20df04f1d024171e136ecd823efcf108785d63883a75b977dfe33"}
Nov 03 21:55:01.579887 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.579865    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" event={"ID":"4fda315c-f98f-4687-8c2e-e01f4f293891","Type":"ContainerStarted","Data":"b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54"}
Nov 03 21:55:01.580008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.579912    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:01.580008 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.579928    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" event={"ID":"4fda315c-f98f-4687-8c2e-e01f4f293891","Type":"ContainerStarted","Data":"0a2cf742aba6f57459acb0229bd8152f8b4c7df225667951e532df41a079a5b7"}
Nov 03 21:55:01.691306 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:01.691246    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" podStartSLOduration=1.691231625 podStartE2EDuration="1.691231625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:01.689578318 +0000 UTC m=+2372.673574896" watchObservedRunningTime="2026-11-03 21:55:01.691231625 +0000 UTC m=+2372.675228183"
Nov 03 21:55:12.589074 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:12.589045    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 21:55:12.589537 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:12.589127    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 21:55:12.704054 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:12.703995    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" podStartSLOduration=12.703980049 podStartE2EDuration="12.703980049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:01.810350402 +0000 UTC m=+2372.794346984" watchObservedRunningTime="2026-11-03 21:55:12.703980049 +0000 UTC m=+2383.687976655"
Nov 03 21:55:22.437623 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:55:22.437580    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:55:29.485316 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.485268    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"]
Nov 03 21:55:29.485934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.485592    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6" podUID="d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" containerName="payload-processing" containerID="cri-o://9ad10660629c55a0b723f5349ec1d9a725bf32ebced41107f630952dfc2cf82b" gracePeriod=30
Nov 03 21:55:29.762256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.762220    5244 generic.go:356] "Generic (PLEG): container finished" podID="d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" containerID="9ad10660629c55a0b723f5349ec1d9a725bf32ebced41107f630952dfc2cf82b" exitCode=0
Nov 03 21:55:29.762440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.762289    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6" event={"ID":"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1","Type":"ContainerDied","Data":"9ad10660629c55a0b723f5349ec1d9a725bf32ebced41107f630952dfc2cf82b"}
Nov 03 21:55:29.762488 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.762456    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6" event={"ID":"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1","Type":"ContainerDied","Data":"8e4185086d69d8b4b930adad238349fcfa5b4a350d3246780c1ebbb4bb633155"}
Nov 03 21:55:29.762488 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.762482    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e4185086d69d8b4b930adad238349fcfa5b4a350d3246780c1ebbb4bb633155"
Nov 03 21:55:29.766434 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.766411    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:55:29.876644 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.876613    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd\" (UniqueName: \"kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd\") pod \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") "
Nov 03 21:55:29.876824 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.876703    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume\") pod \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\" (UID: \"d99f73bc-8f0e-4778-bed8-cfae7ed9dce1\") "
Nov 03 21:55:29.877096 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.877071    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume" pod "d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" (UID: "d99f73bc-8f0e-4778-bed8-cfae7ed9dce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:55:29.879058 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.879030    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd" pod "d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" (UID: "d99f73bc-8f0e-4778-bed8-cfae7ed9dce1"). InnerVolumeSpecName "kube-api-access-tbphd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:55:29.977608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.977566    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:29.977608 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:29.977601    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbphd\" (UniqueName: \"kubernetes.io/projected/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1-kube-api-access-tbphd\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:55:30.768153 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:30.768108    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"
Nov 03 21:55:30.935523 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:30.935488    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"]
Nov 03 21:55:30.939082 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:30.939047    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1c3bed-5cfbf5bf78-sjnc6"]
Nov 03 21:55:31.136605 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:31.136515    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" path="/var/lib/kubelet/pods/d99f73bc-8f0e-4778-bed8-cfae7ed9dce1/volumes"
Nov 03 21:55:31.327900 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:31.327864    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"]
Nov 03 21:55:31.330448 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:31.330421    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1c3bed-m22c5"]
Nov 03 21:55:33.135931 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:33.135888    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059e07b1-ee7e-498e-8ba5-9a2a79152f31" path="/var/lib/kubelet/pods/059e07b1-ee7e-498e-8ba5-9a2a79152f31/volumes"
Nov 03 21:55:40.698567 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.698521    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"]
Nov 03 21:55:40.699167 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.699144    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" containerName="payload-processing"
Nov 03 21:55:40.699255 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.699171    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" containerName="payload-processing"
Nov 03 21:55:40.699365 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.699349    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99f73bc-8f0e-4778-bed8-cfae7ed9dce1" containerName="payload-processing"
Nov 03 21:55:40.703838 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.703812    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:40.758105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.758042    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cd2a9f\""
Nov 03 21:55:40.814284 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.814240    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cd2a9f-dockercfg-wlqmj\""
Nov 03 21:55:40.816294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.816268    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"]
Nov 03 21:55:40.819882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.819700    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnls9\" (UniqueName: \"kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9\") pod \"payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:40.819882 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.819816    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:40.869660 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.869627    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"]
Nov 03 21:55:40.873573 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.873547    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:40.921802 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.921767    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnls9\" (UniqueName: \"kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9\") pod \"payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:40.921991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.921858    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:40.921991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.921916    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:40.921991 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.921958    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnqsd\" (UniqueName: \"kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd\") pod \"payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:40.922595 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.922575    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:40.930299 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.930250    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"]
Nov 03 21:55:40.979396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:40.979277    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnls9\" (UniqueName: \"kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9\") pod \"payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:41.016931 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.016893    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:41.022637 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.022613    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:41.022784 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.022650    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnqsd\" (UniqueName: \"kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd\") pod \"payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:41.023476 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.023450    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:41.086734 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.080962    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnqsd\" (UniqueName: \"kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd\") pod \"payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:41.188101 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.188065    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:41.394791 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.394767    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"]
Nov 03 21:55:41.397479 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:55:41.397445    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265ae3b4_30c0_494a_8f39_e69127811c1a.slice/crio-dac7544d0bb3bc7a2e8ed8c4e54127aa4c57229da3929195210d1d65fedef869 WatchSource:0}: Error finding container dac7544d0bb3bc7a2e8ed8c4e54127aa4c57229da3929195210d1d65fedef869: Status 404 returned error can't find the container with id dac7544d0bb3bc7a2e8ed8c4e54127aa4c57229da3929195210d1d65fedef869
Nov 03 21:55:41.592951 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.592911    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"]
Nov 03 21:55:41.610345 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.610303    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:55:41.613824 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:55:41.613789    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82737aa8_8771_48b4_a627_4427949aa9b3.slice/crio-94ec7fc51ef38101647ab17a484bdbf1fca250deeecaae3c3332f4f9401cfda6 WatchSource:0}: Error finding container 94ec7fc51ef38101647ab17a484bdbf1fca250deeecaae3c3332f4f9401cfda6: Status 404 returned error can't find the container with id 94ec7fc51ef38101647ab17a484bdbf1fca250deeecaae3c3332f4f9401cfda6
Nov 03 21:55:41.834856 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.834752    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq" event={"ID":"82737aa8-8771-48b4-a627-4427949aa9b3","Type":"ContainerStarted","Data":"9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa"}
Nov 03 21:55:41.834856 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.834811    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq" event={"ID":"82737aa8-8771-48b4-a627-4427949aa9b3","Type":"ContainerStarted","Data":"94ec7fc51ef38101647ab17a484bdbf1fca250deeecaae3c3332f4f9401cfda6"}
Nov 03 21:55:41.835412 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.834898    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:55:41.836364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.836340    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" event={"ID":"265ae3b4-30c0-494a-8f39-e69127811c1a","Type":"ContainerStarted","Data":"e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa"}
Nov 03 21:55:41.836461 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.836381    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" event={"ID":"265ae3b4-30c0-494a-8f39-e69127811c1a","Type":"ContainerStarted","Data":"dac7544d0bb3bc7a2e8ed8c4e54127aa4c57229da3929195210d1d65fedef869"}
Nov 03 21:55:41.836461 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.836411    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:41.948710 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:41.948656    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq" podStartSLOduration=1.9486429379999999 podStartE2EDuration="1.948642938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:41.948108733 +0000 UTC m=+2412.932105318" watchObservedRunningTime="2026-11-03 21:55:41.948642938 +0000 UTC m=+2412.932639496"
Nov 03 21:55:42.066353 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:42.065054    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" podStartSLOduration=2.065032654 podStartE2EDuration="2.065032654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:55:42.062957032 +0000 UTC m=+2413.046953592" watchObservedRunningTime="2026-11-03 21:55:42.065032654 +0000 UTC m=+2413.049029215"
Nov 03 21:55:52.436836 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:55:52.436796    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:55:52.845250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:52.845161    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:55:52.845250 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:55:52.845238    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:56:13.823679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:13.823582    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:56:22.436941 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:56:22.436898    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:56:26.019018 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.018981    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"]
Nov 03 21:56:26.022694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.022671    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.077171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.077126    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:56:26.077440 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.077373    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:56:26.079676 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.079650    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"]
Nov 03 21:56:26.148506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.148472    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.148681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.148594    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcms\" (UniqueName: \"kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.148681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.148622    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.249954 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.249921    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.250129 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.250024    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcms\" (UniqueName: \"kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.250129 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.250053    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.250647 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.250619    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.252484 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.252467    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.306819 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.306736    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcms\" (UniqueName: \"kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms\") pod \"maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.337285 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.337256    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:26.711773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:26.711741    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"]
Nov 03 21:56:26.718145 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:56:26.718111    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33b3311_6d14_4bd7_9117_30c03b4f5a35.slice/crio-30f09be7283c72a849cd67ef023391c847d71dec76d7769fb4938673a59e99b3 WatchSource:0}: Error finding container 30f09be7283c72a849cd67ef023391c847d71dec76d7769fb4938673a59e99b3: Status 404 returned error can't find the container with id 30f09be7283c72a849cd67ef023391c847d71dec76d7769fb4938673a59e99b3
Nov 03 21:56:27.105055 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:27.105015    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc" event={"ID":"d33b3311-6d14-4bd7-9117-30c03b4f5a35","Type":"ContainerStarted","Data":"ada473276d10afd74cb9d2fa49ef308e16c7d46a0de28931ba811115c04b332f"}
Nov 03 21:56:27.105527 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:27.105080    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc" event={"ID":"d33b3311-6d14-4bd7-9117-30c03b4f5a35","Type":"ContainerStarted","Data":"30f09be7283c72a849cd67ef023391c847d71dec76d7769fb4938673a59e99b3"}
Nov 03 21:56:27.217801 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:27.217734    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc" podStartSLOduration=2.217700567 podStartE2EDuration="2.217700567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:27.21744742 +0000 UTC m=+2458.201444007" watchObservedRunningTime="2026-11-03 21:56:27.217700567 +0000 UTC m=+2458.201697128"
Nov 03 21:56:28.112448 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:28.112406    5244 generic.go:356] "Generic (PLEG): container finished" podID="d33b3311-6d14-4bd7-9117-30c03b4f5a35" containerID="ada473276d10afd74cb9d2fa49ef308e16c7d46a0de28931ba811115c04b332f" exitCode=0
Nov 03 21:56:28.112895 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:28.112495    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc" event={"ID":"d33b3311-6d14-4bd7-9117-30c03b4f5a35","Type":"ContainerDied","Data":"ada473276d10afd74cb9d2fa49ef308e16c7d46a0de28931ba811115c04b332f"}
Nov 03 21:56:29.270237 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.270207    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:29.386463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.386380    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms\" (UniqueName: \"kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms\") pod \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") "
Nov 03 21:56:29.386463 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.386444    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca\") pod \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") "
Nov 03 21:56:29.386711 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.386508    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca\") pod \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\" (UID: \"d33b3311-6d14-4bd7-9117-30c03b4f5a35\") "
Nov 03 21:56:29.387069 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.387046    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca" pod "d33b3311-6d14-4bd7-9117-30c03b4f5a35" (UID: "d33b3311-6d14-4bd7-9117-30c03b4f5a35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:29.389004 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.388981    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca" pod "d33b3311-6d14-4bd7-9117-30c03b4f5a35" (UID: "d33b3311-6d14-4bd7-9117-30c03b4f5a35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:56:29.389538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.389521    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms" pod "d33b3311-6d14-4bd7-9117-30c03b4f5a35" (UID: "d33b3311-6d14-4bd7-9117-30c03b4f5a35"). InnerVolumeSpecName "kube-api-access-kxcms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:29.488227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.488188    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxcms\" (UniqueName: \"kubernetes.io/projected/d33b3311-6d14-4bd7-9117-30c03b4f5a35-kube-api-access-kxcms\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.488227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.488224    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.488227 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.488235    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d33b3311-6d14-4bd7-9117-30c03b4f5a35-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:29.730258 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.730227    5244 scope.go:122] "RemoveContainer" containerID="9ad10660629c55a0b723f5349ec1d9a725bf32ebced41107f630952dfc2cf82b"
Nov 03 21:56:29.739538 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:29.739516    5244 scope.go:122] "RemoveContainer" containerID="fe96b3b38176af177a995fcd836f0cc3145cb58ec8f97b36c149230a436d0fbc"
Nov 03 21:56:30.138673 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:30.138564    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc" event={"ID":"d33b3311-6d14-4bd7-9117-30c03b4f5a35","Type":"ContainerDied","Data":"30f09be7283c72a849cd67ef023391c847d71dec76d7769fb4938673a59e99b3"}
Nov 03 21:56:30.138673 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:30.138627    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f09be7283c72a849cd67ef023391c847d71dec76d7769fb4938673a59e99b3"
Nov 03 21:56:30.138881 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:30.138692    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"
Nov 03 21:56:34.544742 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.544704    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"]
Nov 03 21:56:34.545203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.545155    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d33b3311-6d14-4bd7-9117-30c03b4f5a35" containerName="revoke-keys"
Nov 03 21:56:34.545203 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.545167    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33b3311-6d14-4bd7-9117-30c03b4f5a35" containerName="revoke-keys"
Nov 03 21:56:34.545296 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.545250    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33b3311-6d14-4bd7-9117-30c03b4f5a35" containerName="revoke-keys"
Nov 03 21:56:34.550025 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.550004    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.649907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.649876    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.649907 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.649910    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfsj5\" (UniqueName: \"kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.650127 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.650020    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.652297 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.652276    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:56:34.652364 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.652321    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:56:34.654609 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.654585    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"]
Nov 03 21:56:34.751465 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.751425    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.751649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.751577    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.751649 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.751602    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfsj5\" (UniqueName: \"kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.752148 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.752130    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.754052 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.754034    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.808066 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.807987    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfsj5\" (UniqueName: \"kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:34.863157 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:34.863125    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:35.237033 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:35.237008    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"]
Nov 03 21:56:35.239893 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:56:35.239865    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c39acf8_af18_4295_8503_f7edf9211d10.slice/crio-9148a78742298aab72c2ebf6ab751756d4a991c4a1825919796455619d69359c WatchSource:0}: Error finding container 9148a78742298aab72c2ebf6ab751756d4a991c4a1825919796455619d69359c: Status 404 returned error can't find the container with id 9148a78742298aab72c2ebf6ab751756d4a991c4a1825919796455619d69359c
Nov 03 21:56:36.177837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:36.177795    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77" event={"ID":"6c39acf8-af18-4295-8503-f7edf9211d10","Type":"ContainerStarted","Data":"c1a822cde5ba11130f1174a7ff3942c2803c58c33d9a78817ec586b624c7bb44"}
Nov 03 21:56:36.178253 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:36.177867    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77" event={"ID":"6c39acf8-af18-4295-8503-f7edf9211d10","Type":"ContainerStarted","Data":"9148a78742298aab72c2ebf6ab751756d4a991c4a1825919796455619d69359c"}
Nov 03 21:56:36.295796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:36.295738    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77" podStartSLOduration=2.2957247499999998 podStartE2EDuration="2.29572475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:56:36.294071829 +0000 UTC m=+2467.278068442" watchObservedRunningTime="2026-11-03 21:56:36.29572475 +0000 UTC m=+2467.279721674"
Nov 03 21:56:37.185664 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:37.185625    5244 generic.go:356] "Generic (PLEG): container finished" podID="6c39acf8-af18-4295-8503-f7edf9211d10" containerID="c1a822cde5ba11130f1174a7ff3942c2803c58c33d9a78817ec586b624c7bb44" exitCode=0
Nov 03 21:56:37.186082 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:37.185670    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77" event={"ID":"6c39acf8-af18-4295-8503-f7edf9211d10","Type":"ContainerDied","Data":"c1a822cde5ba11130f1174a7ff3942c2803c58c33d9a78817ec586b624c7bb44"}
Nov 03 21:56:38.340084 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.340050    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:38.490262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.490163    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5\" (UniqueName: \"kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5\") pod \"6c39acf8-af18-4295-8503-f7edf9211d10\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") "
Nov 03 21:56:38.490262 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.490256    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca\") pod \"6c39acf8-af18-4295-8503-f7edf9211d10\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") "
Nov 03 21:56:38.490522 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.490283    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca\") pod \"6c39acf8-af18-4295-8503-f7edf9211d10\" (UID: \"6c39acf8-af18-4295-8503-f7edf9211d10\") "
Nov 03 21:56:38.490722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.490700    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca" pod "6c39acf8-af18-4295-8503-f7edf9211d10" (UID: "6c39acf8-af18-4295-8503-f7edf9211d10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:56:38.492487 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.492463    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5" pod "6c39acf8-af18-4295-8503-f7edf9211d10" (UID: "6c39acf8-af18-4295-8503-f7edf9211d10"). InnerVolumeSpecName "kube-api-access-gfsj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:56:38.492568 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.492488    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca" pod "6c39acf8-af18-4295-8503-f7edf9211d10" (UID: "6c39acf8-af18-4295-8503-f7edf9211d10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:56:38.591467 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.591422    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfsj5\" (UniqueName: \"kubernetes.io/projected/6c39acf8-af18-4295-8503-f7edf9211d10-kube-api-access-gfsj5\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:38.591467 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.591458    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:38.591467 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:38.591469    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c39acf8-af18-4295-8503-f7edf9211d10-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:56:39.197295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:39.197264    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"
Nov 03 21:56:39.197295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:39.197276    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77" event={"ID":"6c39acf8-af18-4295-8503-f7edf9211d10","Type":"ContainerDied","Data":"9148a78742298aab72c2ebf6ab751756d4a991c4a1825919796455619d69359c"}
Nov 03 21:56:39.197492 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:39.197340    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9148a78742298aab72c2ebf6ab751756d4a991c4a1825919796455619d69359c"
Nov 03 21:56:49.193347 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:49.193303    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"]
Nov 03 21:56:49.234638 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:49.234601    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-204763-nvlwc"]
Nov 03 21:56:51.135600 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:51.135519    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33b3311-6d14-4bd7-9117-30c03b4f5a35" path="/var/lib/kubelet/pods/d33b3311-6d14-4bd7-9117-30c03b4f5a35/volumes"
Nov 03 21:56:52.436401 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:56:52.436361    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:56:55.920783 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:55.920749    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"]
Nov 03 21:56:55.921436 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:55.921014    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" containerID="cri-o://fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af" gracePeriod=30
Nov 03 21:56:57.247733 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:57.247692    5244 patch_prober.go:29] interesting pod/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:56:57.248146 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:56:57.247763    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:57:01.182385 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.182358    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:57:01.247841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.247752    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248001 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.247849    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248001 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.247895    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248001 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.247927    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248001 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.247988    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248037    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248061    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248085    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248105    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token\") pod \"e145f8d4-9961-4400-9e8c-587b7084e5d5\" (UID: \"e145f8d4-9961-4400-9e8c-587b7084e5d5\") "
Nov 03 21:57:01.248256 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248190    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:01.248539 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248367    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:01.248539 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248386    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:01.248641 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248546    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.248641 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248566    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.248641 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248580    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.248764 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248649    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:01.248764 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.248554    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:01.250496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.250472    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:57:01.250590 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.250503    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:01.250710 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.250686    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "kube-api-access-kpp97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:01.250815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.250742    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy" pod "e145f8d4-9961-4400-9e8c-587b7084e5d5" (UID: "e145f8d4-9961-4400-9e8c-587b7084e5d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:01.349295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.349238    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp97\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-kube-api-access-kpp97\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.349295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.349281    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.349295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.349293    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.349295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.349308    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e145f8d4-9961-4400-9e8c-587b7084e5d5-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.349295 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.349320    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e145f8d4-9961-4400-9e8c-587b7084e5d5-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.349923 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.349448    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e145f8d4-9961-4400-9e8c-587b7084e5d5-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:01.399501 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.399462    5244 generic.go:356] "Generic (PLEG): container finished" podID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerID="fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af" exitCode=0
Nov 03 21:57:01.399679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.399521    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"
Nov 03 21:57:01.399679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.399548    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" event={"ID":"e145f8d4-9961-4400-9e8c-587b7084e5d5","Type":"ContainerDied","Data":"fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af"}
Nov 03 21:57:01.399679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.399603    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw" event={"ID":"e145f8d4-9961-4400-9e8c-587b7084e5d5","Type":"ContainerDied","Data":"e06886020e7f67b39de30903e91442755996aeb0f9c353bb87f76deb020cb0d6"}
Nov 03 21:57:01.399679 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.399626    5244 scope.go:122] "RemoveContainer" containerID="fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af"
Nov 03 21:57:01.410046 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.410023    5244 scope.go:122] "RemoveContainer" containerID="0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"
Nov 03 21:57:01.418543 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.418523    5244 scope.go:122] "RemoveContainer" containerID="fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af"
Nov 03 21:57:01.418812 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:57:01.418788    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af\": container with ID starting with fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af not found: ID does not exist" containerID="fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af"
Nov 03 21:57:01.418864 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.418821    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af"} err="failed to get container status \"fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af\": rpc error: code = NotFound desc = could not find container \"fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af\": container with ID starting with fd7e842e4cdf83be55db7188a901d5ab60cdf0b0da39e7935805f06acaa713af not found: ID does not exist"
Nov 03 21:57:01.418864 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.418841    5244 scope.go:122] "RemoveContainer" containerID="0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"
Nov 03 21:57:01.419111 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:57:01.419094    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9\": container with ID starting with 0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9 not found: ID does not exist" containerID="0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"
Nov 03 21:57:01.419174 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.419121    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9"} err="failed to get container status \"0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9\": rpc error: code = NotFound desc = could not find container \"0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9\": container with ID starting with 0e9028e3a5fd6864d67a1635434b92c261949dd7f3167784f53405a972b3bed9 not found: ID does not exist"
Nov 03 21:57:01.565865 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.565781    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"]
Nov 03 21:57:01.570468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:01.570442    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-204763-openshift-default-8589567dd9-gkkvw"]
Nov 03 21:57:03.135733 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:03.135696    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" path="/var/lib/kubelet/pods/e145f8d4-9961-4400-9e8c-587b7084e5d5/volumes"
Nov 03 21:57:08.069170 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:08.069131    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:57:12.797309 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797269    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"]
Nov 03 21:57:12.797780 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797763    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy"
Nov 03 21:57:12.797825 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797782    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy"
Nov 03 21:57:12.797825 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797795    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6c39acf8-af18-4295-8503-f7edf9211d10" containerName="revoke-keys"
Nov 03 21:57:12.797825 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797800    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c39acf8-af18-4295-8503-f7edf9211d10" containerName="revoke-keys"
Nov 03 21:57:12.797913 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797906    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy"
Nov 03 21:57:12.797949 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.797916    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c39acf8-af18-4295-8503-f7edf9211d10" containerName="revoke-keys"
Nov 03 21:57:12.802496 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.802476    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.886452 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.886416    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.886616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.886465    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4j2\" (UniqueName: \"kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.886616 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.886495    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.905063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.905013    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:57:12.905309 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.905145    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:57:12.907169 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.907143    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"]
Nov 03 21:57:12.987344 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.987300    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.987529 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.987390    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4j2\" (UniqueName: \"kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.987529 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.987439    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.988014 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.987985    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:12.989934 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:12.989913    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:13.047544 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:13.047468    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4j2\" (UniqueName: \"kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2\") pod \"maas-api-revoke-sub-848b7cd6-tbv5q\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:13.115709 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:13.115680    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:13.491722 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:13.491695    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"]
Nov 03 21:57:13.493685 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:57:13.493654    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8115f0_811d_4085_b736_651b8c0f9309.slice/crio-b094b40f42c593a1579931f8ae3ab60914aa4e4a09635b940bb9c9d680604d7b WatchSource:0}: Error finding container b094b40f42c593a1579931f8ae3ab60914aa4e4a09635b940bb9c9d680604d7b: Status 404 returned error can't find the container with id b094b40f42c593a1579931f8ae3ab60914aa4e4a09635b940bb9c9d680604d7b
Nov 03 21:57:14.476488 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:14.476448    5244 generic.go:356] "Generic (PLEG): container finished" podID="7d8115f0-811d-4085-b736-651b8c0f9309" containerID="e6db2ae4acdbe3dc1240a916407483d1f02b0956ea6de4ace3deb0a7664512f8" exitCode=0
Nov 03 21:57:14.476899 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:14.476507    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q" event={"ID":"7d8115f0-811d-4085-b736-651b8c0f9309","Type":"ContainerDied","Data":"e6db2ae4acdbe3dc1240a916407483d1f02b0956ea6de4ace3deb0a7664512f8"}
Nov 03 21:57:14.476899 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:14.476572    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q" event={"ID":"7d8115f0-811d-4085-b736-651b8c0f9309","Type":"ContainerStarted","Data":"b094b40f42c593a1579931f8ae3ab60914aa4e4a09635b940bb9c9d680604d7b"}
Nov 03 21:57:15.642806 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.642779    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:15.718253 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.718221    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca\") pod \"7d8115f0-811d-4085-b736-651b8c0f9309\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") "
Nov 03 21:57:15.718491 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.718282    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2\" (UniqueName: \"kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2\") pod \"7d8115f0-811d-4085-b736-651b8c0f9309\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") "
Nov 03 21:57:15.718491 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.718309    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca\") pod \"7d8115f0-811d-4085-b736-651b8c0f9309\" (UID: \"7d8115f0-811d-4085-b736-651b8c0f9309\") "
Nov 03 21:57:15.718626 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.718607    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca" pod "7d8115f0-811d-4085-b736-651b8c0f9309" (UID: "7d8115f0-811d-4085-b736-651b8c0f9309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:57:15.720630 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.720601    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca" pod "7d8115f0-811d-4085-b736-651b8c0f9309" (UID: "7d8115f0-811d-4085-b736-651b8c0f9309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:15.720724 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.720601    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2" pod "7d8115f0-811d-4085-b736-651b8c0f9309" (UID: "7d8115f0-811d-4085-b736-651b8c0f9309"). InnerVolumeSpecName "kube-api-access-zl4j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:15.819958 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.819866    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:15.819958 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.819897    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4j2\" (UniqueName: \"kubernetes.io/projected/7d8115f0-811d-4085-b736-651b8c0f9309-kube-api-access-zl4j2\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:15.819958 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:15.819907    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8115f0-811d-4085-b736-651b8c0f9309-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:16.488632 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:16.488600    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"
Nov 03 21:57:16.488795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:16.488628    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q" event={"ID":"7d8115f0-811d-4085-b736-651b8c0f9309","Type":"ContainerDied","Data":"b094b40f42c593a1579931f8ae3ab60914aa4e4a09635b940bb9c9d680604d7b"}
Nov 03 21:57:16.488795 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:16.488682    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b094b40f42c593a1579931f8ae3ab60914aa4e4a09635b940bb9c9d680604d7b"
Nov 03 21:57:22.438032 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:57:22.437991    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:57:22.550210 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:22.550172    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"]
Nov 03 21:57:22.599442 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:22.599400    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca758d-2dp77"]
Nov 03 21:57:23.137126 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:23.137092    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c39acf8-af18-4295-8503-f7edf9211d10" path="/var/lib/kubelet/pods/6c39acf8-af18-4295-8503-f7edf9211d10/volumes"
Nov 03 21:57:29.764506 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:29.764478    5244 scope.go:122] "RemoveContainer" containerID="35a326ee23c15386be12dec748cb73269994925faead05db4b7152bc1b9c7c40"
Nov 03 21:57:37.744661 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:37.744571    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"]
Nov 03 21:57:40.924815 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:40.924780    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:57:52.436365 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:57:52.436308    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:57:52.765252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:52.765152    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" podUID="9a208866-745d-4596-81d3-5426946a2e88" containerName="main" containerID="cri-o://04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15" gracePeriod=45
Nov 03 21:57:52.897622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:52.897571    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" podUID="9a208866-745d-4596-81d3-5426946a2e88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": dial tcp 10.133.0.86:8000: connect: connection refused"
Nov 03 21:57:53.025394 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.025320    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:57:53.157080 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.156668    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm\") pod \"9a208866-745d-4596-81d3-5426946a2e88\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") "
Nov 03 21:57:53.157873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.157638    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home\") pod \"9a208866-745d-4596-81d3-5426946a2e88\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") "
Nov 03 21:57:53.157873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.157747    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555\" (UniqueName: \"kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555\") pod \"9a208866-745d-4596-81d3-5426946a2e88\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") "
Nov 03 21:57:53.157873 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.157783    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir\") pod \"9a208866-745d-4596-81d3-5426946a2e88\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") "
Nov 03 21:57:53.158114 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.157892    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home" pod "9a208866-745d-4596-81d3-5426946a2e88" (UID: "9a208866-745d-4596-81d3-5426946a2e88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:53.158352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.158217    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache\") pod \"9a208866-745d-4596-81d3-5426946a2e88\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") "
Nov 03 21:57:53.158352 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.158303    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs\") pod \"9a208866-745d-4596-81d3-5426946a2e88\" (UID: \"9a208866-745d-4596-81d3-5426946a2e88\") "
Nov 03 21:57:53.159124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.158219    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir" pod "9a208866-745d-4596-81d3-5426946a2e88" (UID: "9a208866-745d-4596-81d3-5426946a2e88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:53.159124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.158527    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache" pod "9a208866-745d-4596-81d3-5426946a2e88" (UID: "9a208866-745d-4596-81d3-5426946a2e88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:53.159124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.159049    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:53.159124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.159073    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:53.159124 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.159086    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:53.163468 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.163433    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm" pod "9a208866-745d-4596-81d3-5426946a2e88" (UID: "9a208866-745d-4596-81d3-5426946a2e88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:57:53.163601 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.163482    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555" pod "9a208866-745d-4596-81d3-5426946a2e88" (UID: "9a208866-745d-4596-81d3-5426946a2e88"). InnerVolumeSpecName "kube-api-access-7g555". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:57:53.165489 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.165457    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs" pod "9a208866-745d-4596-81d3-5426946a2e88" (UID: "9a208866-745d-4596-81d3-5426946a2e88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:57:53.260238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.260200    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g555\" (UniqueName: \"kubernetes.io/projected/9a208866-745d-4596-81d3-5426946a2e88-kube-api-access-7g555\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:53.260238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.260236    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a208866-745d-4596-81d3-5426946a2e88-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:53.260238 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.260249    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a208866-745d-4596-81d3-5426946a2e88-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:57:53.694868 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.694825    5244 generic.go:356] "Generic (PLEG): container finished" podID="9a208866-745d-4596-81d3-5426946a2e88" containerID="04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15" exitCode=0
Nov 03 21:57:53.695313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.694902    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"
Nov 03 21:57:53.695313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.694898    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" event={"ID":"9a208866-745d-4596-81d3-5426946a2e88","Type":"ContainerDied","Data":"04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15"}
Nov 03 21:57:53.695313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.695008    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9" event={"ID":"9a208866-745d-4596-81d3-5426946a2e88","Type":"ContainerDied","Data":"dc4d709eaa857d22912b1a11d9e568ea7095159bff53f542c245b9260108b06b"}
Nov 03 21:57:53.695313 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.695025    5244 scope.go:122] "RemoveContainer" containerID="04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15"
Nov 03 21:57:53.704564 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.704547    5244 scope.go:122] "RemoveContainer" containerID="04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15"
Nov 03 21:57:53.704858 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:57:53.704835    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15\": container with ID starting with 04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15 not found: ID does not exist" containerID="04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15"
Nov 03 21:57:53.704935 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.704869    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15"} err="failed to get container status \"04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15\": rpc error: code = NotFound desc = could not find container \"04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15\": container with ID starting with 04c9fc233875f2fed1ed3f0a50d513b6fcf998ad3d369ae605ecc0304edc4d15 not found: ID does not exist"
Nov 03 21:57:53.857264 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.857220    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"]
Nov 03 21:57:53.862420 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:53.862397    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-87e1cf/tenantcfg-model-87e1cf-kserve-669b9666f7-nnww9"]
Nov 03 21:57:55.137396 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:57:55.137361    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a208866-745d-4596-81d3-5426946a2e88" path="/var/lib/kubelet/pods/9a208866-745d-4596-81d3-5426946a2e88/volumes"
Nov 03 21:58:12.276737 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.276698    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"]
Nov 03 21:58:12.277378 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277344    5244 cpu_manager.go:414] "containerMap: removing container" podUID="9a208866-745d-4596-81d3-5426946a2e88" containerName="main"
Nov 03 21:58:12.277378 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277366    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a208866-745d-4596-81d3-5426946a2e88" containerName="main"
Nov 03 21:58:12.277524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277400    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7d8115f0-811d-4085-b736-651b8c0f9309" containerName="revoke-keys"
Nov 03 21:58:12.277524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277408    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8115f0-811d-4085-b736-651b8c0f9309" containerName="revoke-keys"
Nov 03 21:58:12.277524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277427    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy"
Nov 03 21:58:12.277524 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277434    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy"
Nov 03 21:58:12.277715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277564    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a208866-745d-4596-81d3-5426946a2e88" containerName="main"
Nov 03 21:58:12.277715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277579    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8115f0-811d-4085-b736-651b8c0f9309" containerName="revoke-keys"
Nov 03 21:58:12.277715 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.277590    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e145f8d4-9961-4400-9e8c-587b7084e5d5" containerName="istio-proxy"
Nov 03 21:58:12.281966 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.281944    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.284650 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.284623    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.284755 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.284673    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5b9\" (UniqueName: \"kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9\") pod \"payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.335520 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.335456    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4644ae\""
Nov 03 21:58:12.337654 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.337616    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"]
Nov 03 21:58:12.386796 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.386764    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4644ae-dockercfg-gt6z9\""
Nov 03 21:58:12.414222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.400088    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.414502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.401051    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.414502 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.414261    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5b9\" (UniqueName: \"kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9\") pod \"payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.420374 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.420302    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"]
Nov 03 21:58:12.426693 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.426662    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.493409 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.493372    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"]
Nov 03 21:58:12.498941 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.498914    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5b9\" (UniqueName: \"kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9\") pod \"payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.515550 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.515516    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.515725 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.515583    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6m7\" (UniqueName: \"kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7\") pod \"payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.595165 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.595076    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:12.616494 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.616457    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.616625 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.616543    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6m7\" (UniqueName: \"kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7\") pod \"payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.617147 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.617126    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.673114 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.673076    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6m7\" (UniqueName: \"kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7\") pod \"payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.744969 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.744938    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:12.969963 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:12.969937    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"]
Nov 03 21:58:12.972883 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:58:12.972855    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2adf496_a1d9_4dd5_8ce0_cc6c3131c74d.slice/crio-10c761ac07460ea0338a83032d723ce37b0320f7fde3b099dd77e33c8858b6bf WatchSource:0}: Error finding container 10c761ac07460ea0338a83032d723ce37b0320f7fde3b099dd77e33c8858b6bf: Status 404 returned error can't find the container with id 10c761ac07460ea0338a83032d723ce37b0320f7fde3b099dd77e33c8858b6bf
Nov 03 21:58:13.122712 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.122684    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"]
Nov 03 21:58:13.125446 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:58:13.125412    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49061d70_1848_4253_8a5f_c0899631d52e.slice/crio-c9d2ec27339e8d079d551ad397f76516793cd6fd6e57153d086335b145070f99 WatchSource:0}: Error finding container c9d2ec27339e8d079d551ad397f76516793cd6fd6e57153d086335b145070f99: Status 404 returned error can't find the container with id c9d2ec27339e8d079d551ad397f76516793cd6fd6e57153d086335b145070f99
Nov 03 21:58:13.812222 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.812182    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw" event={"ID":"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d","Type":"ContainerStarted","Data":"e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64"}
Nov 03 21:58:13.812730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.812244    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw" event={"ID":"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d","Type":"ContainerStarted","Data":"10c761ac07460ea0338a83032d723ce37b0320f7fde3b099dd77e33c8858b6bf"}
Nov 03 21:58:13.812730 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.812280    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:13.813959 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.813933    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7" event={"ID":"49061d70-1848-4253-8a5f-c0899631d52e","Type":"ContainerStarted","Data":"2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c"}
Nov 03 21:58:13.814115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.813972    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7" event={"ID":"49061d70-1848-4253-8a5f-c0899631d52e","Type":"ContainerStarted","Data":"c9d2ec27339e8d079d551ad397f76516793cd6fd6e57153d086335b145070f99"}
Nov 03 21:58:13.814115 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.814056    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:13.926674 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:13.926620    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw" podStartSLOduration=1.926604003 podStartE2EDuration="1.926604003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:13.924502185 +0000 UTC m=+2564.908498745" watchObservedRunningTime="2026-11-03 21:58:13.926604003 +0000 UTC m=+2564.910600631"
Nov 03 21:58:14.040010 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:14.039950    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7" podStartSLOduration=2.039936748 podStartE2EDuration="2.039936748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:58:14.03761473 +0000 UTC m=+2565.021611294" watchObservedRunningTime="2026-11-03 21:58:14.039936748 +0000 UTC m=+2565.023933304"
Nov 03 21:58:22.436768 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:58:22.436725    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:58:24.821311 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:24.821279    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 21:58:24.821794 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:24.821396    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 21:58:29.790623 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:29.790593    5244 scope.go:122] "RemoveContainer" containerID="b780e0970bd7dbe73036c26df320ea647661c8a0551e1e3f46dc1fc2a1b4c1fe"
Nov 03 21:58:42.766917 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:58:42.766881    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:58:52.439450 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:58:52.439402    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:59:07.487252 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:07.487171    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"]
Nov 03 21:59:07.597224 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:07.597179    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"]
Nov 03 21:59:12.487689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.487642    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq" podUID="82737aa8-8771-48b4-a627-4427949aa9b3" containerName="payload-processing" containerID="cri-o://9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa" gracePeriod=25
Nov 03 21:59:12.597856 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.597815    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" podUID="265ae3b4-30c0-494a-8f39-e69127811c1a" containerName="payload-pre-processing" containerID="cri-o://e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa" gracePeriod=25
Nov 03 21:59:12.746959 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.746877    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:59:12.809233 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.809190    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd\" (UniqueName: \"kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd\") pod \"82737aa8-8771-48b4-a627-4427949aa9b3\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") "
Nov 03 21:59:12.809439 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.809348    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume\") pod \"82737aa8-8771-48b4-a627-4427949aa9b3\" (UID: \"82737aa8-8771-48b4-a627-4427949aa9b3\") "
Nov 03 21:59:12.809773 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.809742    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume" pod "82737aa8-8771-48b4-a627-4427949aa9b3" (UID: "82737aa8-8771-48b4-a627-4427949aa9b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:59:12.811673 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.811646    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd" pod "82737aa8-8771-48b4-a627-4427949aa9b3" (UID: "82737aa8-8771-48b4-a627-4427949aa9b3"). InnerVolumeSpecName "kube-api-access-hnqsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:12.854543 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.854518    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:59:12.910612 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.910579    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume\") pod \"265ae3b4-30c0-494a-8f39-e69127811c1a\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") "
Nov 03 21:59:12.910812 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.910637    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9\" (UniqueName: \"kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9\") pod \"265ae3b4-30c0-494a-8f39-e69127811c1a\" (UID: \"265ae3b4-30c0-494a-8f39-e69127811c1a\") "
Nov 03 21:59:12.910988 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.910963    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume" pod "265ae3b4-30c0-494a-8f39-e69127811c1a" (UID: "265ae3b4-30c0-494a-8f39-e69127811c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:59:12.911114 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.911098    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265ae3b4-30c0-494a-8f39-e69127811c1a-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:12.911175 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.911117    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnqsd\" (UniqueName: \"kubernetes.io/projected/82737aa8-8771-48b4-a627-4427949aa9b3-kube-api-access-hnqsd\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:12.911175 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.911126    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82737aa8-8771-48b4-a627-4427949aa9b3-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:12.913004 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:12.912987    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9" pod "265ae3b4-30c0-494a-8f39-e69127811c1a" (UID: "265ae3b4-30c0-494a-8f39-e69127811c1a"). InnerVolumeSpecName "kube-api-access-tnls9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:13.012235 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.012156    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnls9\" (UniqueName: \"kubernetes.io/projected/265ae3b4-30c0-494a-8f39-e69127811c1a-kube-api-access-tnls9\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:13.175689 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.175649    5244 generic.go:356] "Generic (PLEG): container finished" podID="82737aa8-8771-48b4-a627-4427949aa9b3" containerID="9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa" exitCode=0
Nov 03 21:59:13.175869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.175712    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"
Nov 03 21:59:13.175869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.175733    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq" event={"ID":"82737aa8-8771-48b4-a627-4427949aa9b3","Type":"ContainerDied","Data":"9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa"}
Nov 03 21:59:13.175869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.175828    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq" event={"ID":"82737aa8-8771-48b4-a627-4427949aa9b3","Type":"ContainerDied","Data":"94ec7fc51ef38101647ab17a484bdbf1fca250deeecaae3c3332f4f9401cfda6"}
Nov 03 21:59:13.175869 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.175859    5244 scope.go:122] "RemoveContainer" containerID="9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa"
Nov 03 21:59:13.177306 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.177286    5244 generic.go:356] "Generic (PLEG): container finished" podID="265ae3b4-30c0-494a-8f39-e69127811c1a" containerID="e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa" exitCode=0
Nov 03 21:59:13.177445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.177360    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" event={"ID":"265ae3b4-30c0-494a-8f39-e69127811c1a","Type":"ContainerDied","Data":"e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa"}
Nov 03 21:59:13.177445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.177390    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" event={"ID":"265ae3b4-30c0-494a-8f39-e69127811c1a","Type":"ContainerDied","Data":"dac7544d0bb3bc7a2e8ed8c4e54127aa4c57229da3929195210d1d65fedef869"}
Nov 03 21:59:13.177445 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.177427    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"
Nov 03 21:59:13.185448 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.185427    5244 scope.go:122] "RemoveContainer" containerID="9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa"
Nov 03 21:59:13.185752 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:59:13.185734    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa\": container with ID starting with 9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa not found: ID does not exist" containerID="9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa"
Nov 03 21:59:13.185800 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.185761    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa"} err="failed to get container status \"9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa\": rpc error: code = NotFound desc = could not find container \"9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa\": container with ID starting with 9d3f30a9c689e4d047e5bdf8037195816e272d84a3556db144c3a355bd24d0fa not found: ID does not exist"
Nov 03 21:59:13.185800 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.185778    5244 scope.go:122] "RemoveContainer" containerID="e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa"
Nov 03 21:59:13.198429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.198406    5244 scope.go:122] "RemoveContainer" containerID="e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa"
Nov 03 21:59:13.198741 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:59:13.198719    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa\": container with ID starting with e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa not found: ID does not exist" containerID="e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa"
Nov 03 21:59:13.198789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.198751    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa"} err="failed to get container status \"e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa\": rpc error: code = NotFound desc = could not find container \"e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa\": container with ID starting with e94e3d9b427694a76ac22031b6d02bb9e29de030e56794e5b9ff0f0dde935eaa not found: ID does not exist"
Nov 03 21:59:13.338068 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.337981    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"]
Nov 03 21:59:13.343841 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.343810    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5"]
Nov 03 21:59:13.451267 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.451230    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"]
Nov 03 21:59:13.454027 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.453998    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd2a9f-788ff6f7bd-qjwjq"]
Nov 03 21:59:13.843789 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:13.843749    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd2a9f-768757f49f-h7lk5" podUID="265ae3b4-30c0-494a-8f39-e69127811c1a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.101:50052\" within 1s: context deadline exceeded"
Nov 03 21:59:13.844194 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:59:13.843882    5244 logging.go:55] [core] [Channel #827 SubChannel #828]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.101:50052", ServerName: "10.133.0.101:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.101:50052: operation was canceled"
Nov 03 21:59:15.136818 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:15.136777    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265ae3b4-30c0-494a-8f39-e69127811c1a" path="/var/lib/kubelet/pods/265ae3b4-30c0-494a-8f39-e69127811c1a/volumes"
Nov 03 21:59:15.137369 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:15.137308    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82737aa8-8771-48b4-a627-4427949aa9b3" path="/var/lib/kubelet/pods/82737aa8-8771-48b4-a627-4427949aa9b3/volumes"
Nov 03 21:59:19.420211 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420172    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"]
Nov 03 21:59:19.420705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420589    5244 cpu_manager.go:414] "containerMap: removing container" podUID="265ae3b4-30c0-494a-8f39-e69127811c1a" containerName="payload-pre-processing"
Nov 03 21:59:19.420705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420601    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265ae3b4-30c0-494a-8f39-e69127811c1a" containerName="payload-pre-processing"
Nov 03 21:59:19.420705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420614    5244 cpu_manager.go:414] "containerMap: removing container" podUID="82737aa8-8771-48b4-a627-4427949aa9b3" containerName="payload-processing"
Nov 03 21:59:19.420705 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420621    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82737aa8-8771-48b4-a627-4427949aa9b3" containerName="payload-processing"
Nov 03 21:59:19.420840 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420711    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="265ae3b4-30c0-494a-8f39-e69127811c1a" containerName="payload-pre-processing"
Nov 03 21:59:19.420840 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.420720    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="82737aa8-8771-48b4-a627-4427949aa9b3" containerName="payload-processing"
Nov 03 21:59:19.425428 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.425407    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.477318 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.477279    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.477537 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.477400    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.477537 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.477494    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzd4t\" (UniqueName: \"kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.481062 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.481034    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"]
Nov 03 21:59:19.528411 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.528379    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:59:19.528579 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.528469    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:59:19.578743 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.578708    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzd4t\" (UniqueName: \"kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.578908 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.578811    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.578908 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.578848    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.579480 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.579462    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.581294 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.581275    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.637469 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.637444    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzd4t\" (UniqueName: \"kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t\") pod \"maas-api-revoke-sub-b768a57a-cbpv7\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:19.738371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:19.738316    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:20.114125 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:20.114102    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"]
Nov 03 21:59:20.116749 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:59:20.116719    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93a5403_498b_4431_b5bc_ee35b291f889.slice/crio-58e6a570640e3005470a198f1c32f0682cc9482d92daf7e540c07f839fd9c1e1 WatchSource:0}: Error finding container 58e6a570640e3005470a198f1c32f0682cc9482d92daf7e540c07f839fd9c1e1: Status 404 returned error can't find the container with id 58e6a570640e3005470a198f1c32f0682cc9482d92daf7e540c07f839fd9c1e1
Nov 03 21:59:20.216198 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:20.216159    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7" event={"ID":"d93a5403-498b-4431-b5bc-ee35b291f889","Type":"ContainerStarted","Data":"58e6a570640e3005470a198f1c32f0682cc9482d92daf7e540c07f839fd9c1e1"}
Nov 03 21:59:21.223922 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:21.223888    5244 generic.go:356] "Generic (PLEG): container finished" podID="d93a5403-498b-4431-b5bc-ee35b291f889" containerID="92857abca0a7e55de7a4f88f9d53ad3e606939e1866bd9cbdb6edb9b7cab734d" exitCode=0
Nov 03 21:59:21.224373 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:21.223962    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7" event={"ID":"d93a5403-498b-4431-b5bc-ee35b291f889","Type":"ContainerDied","Data":"92857abca0a7e55de7a4f88f9d53ad3e606939e1866bd9cbdb6edb9b7cab734d"}
Nov 03 21:59:22.369078 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.369047    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:22.409622 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.409579    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t\" (UniqueName: \"kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t\") pod \"d93a5403-498b-4431-b5bc-ee35b291f889\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") "
Nov 03 21:59:22.409836 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.409651    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca\") pod \"d93a5403-498b-4431-b5bc-ee35b291f889\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") "
Nov 03 21:59:22.409836 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.409694    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca\") pod \"d93a5403-498b-4431-b5bc-ee35b291f889\" (UID: \"d93a5403-498b-4431-b5bc-ee35b291f889\") "
Nov 03 21:59:22.410016 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.409996    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca" pod "d93a5403-498b-4431-b5bc-ee35b291f889" (UID: "d93a5403-498b-4431-b5bc-ee35b291f889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:59:22.411961 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.411929    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca" pod "d93a5403-498b-4431-b5bc-ee35b291f889" (UID: "d93a5403-498b-4431-b5bc-ee35b291f889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:59:22.412071 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.412035    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t" pod "d93a5403-498b-4431-b5bc-ee35b291f889" (UID: "d93a5403-498b-4431-b5bc-ee35b291f889"). InnerVolumeSpecName "kube-api-access-lzd4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:22.437358 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:59:22.437303    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:59:22.511837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.511737    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzd4t\" (UniqueName: \"kubernetes.io/projected/d93a5403-498b-4431-b5bc-ee35b291f889-kube-api-access-lzd4t\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:22.511837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.511780    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:22.511837 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:22.511797    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d93a5403-498b-4431-b5bc-ee35b291f889-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:23.236093 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:23.236065    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"
Nov 03 21:59:23.236292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:23.236061    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7" event={"ID":"d93a5403-498b-4431-b5bc-ee35b291f889","Type":"ContainerDied","Data":"58e6a570640e3005470a198f1c32f0682cc9482d92daf7e540c07f839fd9c1e1"}
Nov 03 21:59:23.236292 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:23.236203    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58e6a570640e3005470a198f1c32f0682cc9482d92daf7e540c07f839fd9c1e1"
Nov 03 21:59:28.044900 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:28.044864    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"]
Nov 03 21:59:28.094167 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:28.094138    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd1c0a7c-q4vs8"]
Nov 03 21:59:29.136083 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:29.136044    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed61697-2be1-4c26-8cf2-9e20535dab3e" path="/var/lib/kubelet/pods/1ed61697-2be1-4c26-8cf2-9e20535dab3e/volumes"
Nov 03 21:59:29.817368 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:29.817317    5244 scope.go:122] "RemoveContainer" containerID="cee7e5112b01a2ecd0c1d52162fcf23b490ab9afd9d6e907016aeb69b92fa559"
Nov 03 21:59:36.107753 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.107717    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"]
Nov 03 21:59:36.108171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.108150    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d93a5403-498b-4431-b5bc-ee35b291f889" containerName="revoke-keys"
Nov 03 21:59:36.108171 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.108162    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93a5403-498b-4431-b5bc-ee35b291f889" containerName="revoke-keys"
Nov 03 21:59:36.108270 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.108259    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93a5403-498b-4431-b5bc-ee35b291f889" containerName="revoke-keys"
Nov 03 21:59:36.113692 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.113671    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.215268 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.215230    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:59:36.215474 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.215278    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:59:36.217104 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.217084    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"]
Nov 03 21:59:36.260973 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.260939    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.261154 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.261000    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.261213 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.261166    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmnj\" (UniqueName: \"kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.362371 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.362259    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmnj\" (UniqueName: \"kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.362577 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.362556    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.362681 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.362660    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.363457 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.363429    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.365875 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.365852    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.419063 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.419035    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmnj\" (UniqueName: \"kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj\") pod \"maas-api-revoke-sub-368f0084-7kqwc\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.425925 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.425903    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:36.801077 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:36.800954    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"]
Nov 03 21:59:36.804475 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:59:36.804435    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563b243d_b8aa_49a7_baf8_7ec92852cb23.slice/crio-6007e201d19d72cc42aecb6c201fc8a2420c8c8c95bbece0533ec974a72364c5 WatchSource:0}: Error finding container 6007e201d19d72cc42aecb6c201fc8a2420c8c8c95bbece0533ec974a72364c5: Status 404 returned error can't find the container with id 6007e201d19d72cc42aecb6c201fc8a2420c8c8c95bbece0533ec974a72364c5
Nov 03 21:59:37.311623 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:37.311584    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc" event={"ID":"563b243d-b8aa-49a7-baf8-7ec92852cb23","Type":"ContainerStarted","Data":"bdd7755693bfd7ae02b6286a3bdd302f06391a6aeb4626e16ade8d1ea79b9d53"}
Nov 03 21:59:37.312037 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:37.311657    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc" event={"ID":"563b243d-b8aa-49a7-baf8-7ec92852cb23","Type":"ContainerStarted","Data":"6007e201d19d72cc42aecb6c201fc8a2420c8c8c95bbece0533ec974a72364c5"}
Nov 03 21:59:37.424954 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:37.424840    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc" podStartSLOduration=1.424824433 podStartE2EDuration="1.424824433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:37.423104115 +0000 UTC m=+2648.407100693" watchObservedRunningTime="2026-11-03 21:59:37.424824433 +0000 UTC m=+2648.408820988"
Nov 03 21:59:38.318344 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:38.318287    5244 generic.go:356] "Generic (PLEG): container finished" podID="563b243d-b8aa-49a7-baf8-7ec92852cb23" containerID="bdd7755693bfd7ae02b6286a3bdd302f06391a6aeb4626e16ade8d1ea79b9d53" exitCode=0
Nov 03 21:59:38.318761 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:38.318376    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc" event={"ID":"563b243d-b8aa-49a7-baf8-7ec92852cb23","Type":"ContainerDied","Data":"bdd7755693bfd7ae02b6286a3bdd302f06391a6aeb4626e16ade8d1ea79b9d53"}
Nov 03 21:59:39.468914 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.468879    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:39.597926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.597828    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca\") pod \"563b243d-b8aa-49a7-baf8-7ec92852cb23\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") "
Nov 03 21:59:39.598131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.598004    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj\" (UniqueName: \"kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj\") pod \"563b243d-b8aa-49a7-baf8-7ec92852cb23\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") "
Nov 03 21:59:39.598131 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.598061    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca\") pod \"563b243d-b8aa-49a7-baf8-7ec92852cb23\" (UID: \"563b243d-b8aa-49a7-baf8-7ec92852cb23\") "
Nov 03 21:59:39.598429 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.598405    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca" pod "563b243d-b8aa-49a7-baf8-7ec92852cb23" (UID: "563b243d-b8aa-49a7-baf8-7ec92852cb23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:59:39.600388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.600350    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj" pod "563b243d-b8aa-49a7-baf8-7ec92852cb23" (UID: "563b243d-b8aa-49a7-baf8-7ec92852cb23"). InnerVolumeSpecName "kube-api-access-2jmnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:39.600545 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.600449    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca" pod "563b243d-b8aa-49a7-baf8-7ec92852cb23" (UID: "563b243d-b8aa-49a7-baf8-7ec92852cb23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:59:39.699556 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.699461    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:39.699833 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.699808    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jmnj\" (UniqueName: \"kubernetes.io/projected/563b243d-b8aa-49a7-baf8-7ec92852cb23-kube-api-access-2jmnj\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:39.699926 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:39.699914    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563b243d-b8aa-49a7-baf8-7ec92852cb23-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:40.000060 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:40.000021    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 21:59:40.330702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:40.330628    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"
Nov 03 21:59:40.330702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:40.330647    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc" event={"ID":"563b243d-b8aa-49a7-baf8-7ec92852cb23","Type":"ContainerDied","Data":"6007e201d19d72cc42aecb6c201fc8a2420c8c8c95bbece0533ec974a72364c5"}
Nov 03 21:59:40.330702 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:40.330689    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6007e201d19d72cc42aecb6c201fc8a2420c8c8c95bbece0533ec974a72364c5"
Nov 03 21:59:52.178190 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.178150    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"]
Nov 03 21:59:52.178749 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.178614    5244 cpu_manager.go:414] "containerMap: removing container" podUID="563b243d-b8aa-49a7-baf8-7ec92852cb23" containerName="revoke-keys"
Nov 03 21:59:52.178749 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.178626    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563b243d-b8aa-49a7-baf8-7ec92852cb23" containerName="revoke-keys"
Nov 03 21:59:52.178749 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.178696    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="563b243d-b8aa-49a7-baf8-7ec92852cb23" containerName="revoke-keys"
Nov 03 21:59:52.182133 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.182115    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.243798 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.243762    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.243989 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.243928    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.244130 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.244109    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67d2\" (UniqueName: \"kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.285388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.285322    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:59:52.285388 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.285380    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 21:59:52.287442 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.287417    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"]
Nov 03 21:59:52.345283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.345248    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67d2\" (UniqueName: \"kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.345283 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.345287    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.345591 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.345389    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.345995 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.345974    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.347920 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.347899    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.401514 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.401483    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67d2\" (UniqueName: \"kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2\") pod \"maas-api-revoke-sub-ae0ff440-lrfhd\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.435596 ip-10-0-129-65 kubenswrapper[5244]: E1103 21:59:52.435554    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 21:59:52.494558 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.494517    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:52.871665 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:52.871640    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"]
Nov 03 21:59:52.874541 ip-10-0-129-65 kubenswrapper[5244]: W1103 21:59:52.874506    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb35f2d9_5133_4a13_be4c_f35bf89c4d71.slice/crio-6dcbfb16de934730abfa96b0d678a269714269f3e1bfaedb72b2f830f1917ada WatchSource:0}: Error finding container 6dcbfb16de934730abfa96b0d678a269714269f3e1bfaedb72b2f830f1917ada: Status 404 returned error can't find the container with id 6dcbfb16de934730abfa96b0d678a269714269f3e1bfaedb72b2f830f1917ada
Nov 03 21:59:53.410404 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:53.409863    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd" event={"ID":"cb35f2d9-5133-4a13-be4c-f35bf89c4d71","Type":"ContainerStarted","Data":"7d32c040d9f3525eff776d244b0b0d17d4a01d69ea421ed35149d9dd9b9eef44"}
Nov 03 21:59:53.410404 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:53.409942    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd" event={"ID":"cb35f2d9-5133-4a13-be4c-f35bf89c4d71","Type":"ContainerStarted","Data":"6dcbfb16de934730abfa96b0d678a269714269f3e1bfaedb72b2f830f1917ada"}
Nov 03 21:59:53.540096 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:53.540033    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd" podStartSLOduration=1.540012624 podStartE2EDuration="1.540012624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:59:53.538220837 +0000 UTC m=+2664.522217394" watchObservedRunningTime="2026-11-03 21:59:53.540012624 +0000 UTC m=+2664.524009182"
Nov 03 21:59:54.416774 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:54.416739    5244 generic.go:356] "Generic (PLEG): container finished" podID="cb35f2d9-5133-4a13-be4c-f35bf89c4d71" containerID="7d32c040d9f3525eff776d244b0b0d17d4a01d69ea421ed35149d9dd9b9eef44" exitCode=0
Nov 03 21:59:54.417229 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:54.416828    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd" event={"ID":"cb35f2d9-5133-4a13-be4c-f35bf89c4d71","Type":"ContainerDied","Data":"7d32c040d9f3525eff776d244b0b0d17d4a01d69ea421ed35149d9dd9b9eef44"}
Nov 03 21:59:55.572694 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.572666    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:55.683370 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.683317    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2\" (UniqueName: \"kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2\") pod \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") "
Nov 03 21:59:55.683536 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.683412    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca\") pod \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") "
Nov 03 21:59:55.683574 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.683542    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca\") pod \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\" (UID: \"cb35f2d9-5133-4a13-be4c-f35bf89c4d71\") "
Nov 03 21:59:55.683915 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.683888    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca" pod "cb35f2d9-5133-4a13-be4c-f35bf89c4d71" (UID: "cb35f2d9-5133-4a13-be4c-f35bf89c4d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:59:55.685981 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.685953    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2" pod "cb35f2d9-5133-4a13-be4c-f35bf89c4d71" (UID: "cb35f2d9-5133-4a13-be4c-f35bf89c4d71"). InnerVolumeSpecName "kube-api-access-f67d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:59:55.686105 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.686069    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca" pod "cb35f2d9-5133-4a13-be4c-f35bf89c4d71" (UID: "cb35f2d9-5133-4a13-be4c-f35bf89c4d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:59:55.785485 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.785452    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67d2\" (UniqueName: \"kubernetes.io/projected/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-kube-api-access-f67d2\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:55.785485 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.785482    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:55.785485 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:55.785493    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb35f2d9-5133-4a13-be4c-f35bf89c4d71-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 21:59:56.429525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:56.429496    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"
Nov 03 21:59:56.429525 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:56.429511    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd" event={"ID":"cb35f2d9-5133-4a13-be4c-f35bf89c4d71","Type":"ContainerDied","Data":"6dcbfb16de934730abfa96b0d678a269714269f3e1bfaedb72b2f830f1917ada"}
Nov 03 21:59:56.429810 ip-10-0-129-65 kubenswrapper[5244]: I1103 21:59:56.429559    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dcbfb16de934730abfa96b0d678a269714269f3e1bfaedb72b2f830f1917ada"
Nov 03 22:00:00.165039 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.164999    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"]
Nov 03 22:00:00.165517 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.165501    5244 cpu_manager.go:414] "containerMap: removing container" podUID="cb35f2d9-5133-4a13-be4c-f35bf89c4d71" containerName="revoke-keys"
Nov 03 22:00:00.165573 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.165518    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb35f2d9-5133-4a13-be4c-f35bf89c4d71" containerName="revoke-keys"
Nov 03 22:00:00.165611 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.165585    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb35f2d9-5133-4a13-be4c-f35bf89c4d71" containerName="revoke-keys"
Nov 03 22:00:00.168673 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.168656    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"]
Nov 03 22:00:00.168793 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.168773    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.172823 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.172797    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"]
Nov 03 22:00:00.172970 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.172915    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.176681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.176661    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.214845 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.214812    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"]
Nov 03 22:00:00.218239 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.218220    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"]
Nov 03 22:00:00.218382 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.218367    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.221433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.221414    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"]
Nov 03 22:00:00.221548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.221521    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.224603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.224584    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"]
Nov 03 22:00:00.224714 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.224688    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.227975 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.227958    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.245450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.245420    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.245586 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.245566    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.245645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.245617    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.245724 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.245707    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ftg\" (UniqueName: \"kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.245761 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.245748    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vds8t\" (UniqueName: \"kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.245854 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.245838    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llr4c\" (UniqueName: \"kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.261949 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.261925    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:00:00.266973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.266952    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"]
Nov 03 22:00:00.274087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.274063    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"]
Nov 03 22:00:00.306576 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.306237    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"]
Nov 03 22:00:00.333467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.327198    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"]
Nov 03 22:00:00.336380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.334544    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"]
Nov 03 22:00:00.336380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.334619    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"]
Nov 03 22:00:00.346914 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.346882    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.347132 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347098    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.347278 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347135    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.347278 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347177    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ftg\" (UniqueName: \"kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.347278 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347210    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vds8t\" (UniqueName: \"kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.347278 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347247    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-d6dbt\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.347547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347293    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llr4c\" (UniqueName: \"kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.347547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347414    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.347547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347475    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpt9t\" (UniqueName: \"kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t\") pod \"maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.347547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347514    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlx4r\" (UniqueName: \"kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.347763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347567    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.347763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347598    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlv8\" (UniqueName: \"kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8\") pod \"maas-api-key-cleanup-29895720-d6dbt\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.347763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347632    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkslx\" (UniqueName: \"kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx\") pod \"maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.347763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.347668    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.349589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.348423    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.349589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.348846    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.349589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.349540    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.373736 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.373707    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"]
Nov 03 22:00:00.426544 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.426466    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llr4c\" (UniqueName: \"kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.426711 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.426566    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ftg\" (UniqueName: \"kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg\") pod \"maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.426777 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.426753    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vds8t\" (UniqueName: \"kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.448794 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448759    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpt9t\" (UniqueName: \"kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t\") pod \"maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.448990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448816    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlx4r\" (UniqueName: \"kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.448990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448865    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlv8\" (UniqueName: \"kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8\") pod \"maas-api-key-cleanup-29895720-d6dbt\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.448990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448899    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkslx\" (UniqueName: \"kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx\") pod \"maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.448990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448935    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.448990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448963    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.448990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.448988    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.449364 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.449042    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-d6dbt\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.449907 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.449879    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.449907 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.449894    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895720-d6dbt\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.450072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.449927    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.450072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.449973    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.487230 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.487199    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:00.495571 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.495546    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:00.502152 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.502130    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:00.504899 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.504871    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpt9t\" (UniqueName: \"kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t\") pod \"maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.505539 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.505517    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlv8\" (UniqueName: \"kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8\") pod \"maas-api-key-cleanup-29895720-d6dbt\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.533294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.533000    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:00.540936 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.540604    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:00.540936 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.540627    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlx4r\" (UniqueName: \"kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.540936 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.540883    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkslx\" (UniqueName: \"kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx\") pod \"maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.547876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.546884    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:00.554202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.554093    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:00.885921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.885889    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"]
Nov 03 22:00:00.890476 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:00.890449    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a0cea9_bbb2_41b6_9c31_4e66f825f049.slice/crio-ebf2252a15b8326730f309db872a5b0c0edaeb37eedffb7b947a8fc8cda88b2c WatchSource:0}: Error finding container ebf2252a15b8326730f309db872a5b0c0edaeb37eedffb7b947a8fc8cda88b2c: Status 404 returned error can't find the container with id ebf2252a15b8326730f309db872a5b0c0edaeb37eedffb7b947a8fc8cda88b2c
Nov 03 22:00:00.912069 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.912043    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"]
Nov 03 22:00:00.916488 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:00.916461    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc6b000_1c16_43dd_b67f_c24903c56567.slice/crio-e6b1e8663a62b7cba46fccdcd9fec0b0b4582d7fcee1cadfc353e98340c2ca3a WatchSource:0}: Error finding container e6b1e8663a62b7cba46fccdcd9fec0b0b4582d7fcee1cadfc353e98340c2ca3a: Status 404 returned error can't find the container with id e6b1e8663a62b7cba46fccdcd9fec0b0b4582d7fcee1cadfc353e98340c2ca3a
Nov 03 22:00:00.935712 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.935680    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"]
Nov 03 22:00:00.937030 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:00.937006    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0515020_1ea8_4a15_81b3_381ec44b1532.slice/crio-5428427f6c0cb72a6d6888ff93405b3f2707e68bdaad550283a1a5a69d95d577 WatchSource:0}: Error finding container 5428427f6c0cb72a6d6888ff93405b3f2707e68bdaad550283a1a5a69d95d577: Status 404 returned error can't find the container with id 5428427f6c0cb72a6d6888ff93405b3f2707e68bdaad550283a1a5a69d95d577
Nov 03 22:00:00.981580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:00.981548    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"]
Nov 03 22:00:00.985022 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:00.984995    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a630df_7f51_49d7_87d0_d13dd917b535.slice/crio-d7bad2d8cd59afcfb56f2d60f744038b136893d94382df99230e2453e1715f68 WatchSource:0}: Error finding container d7bad2d8cd59afcfb56f2d60f744038b136893d94382df99230e2453e1715f68: Status 404 returned error can't find the container with id d7bad2d8cd59afcfb56f2d60f744038b136893d94382df99230e2453e1715f68
Nov 03 22:00:01.013727 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.013618    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"]
Nov 03 22:00:01.015965 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:01.015911    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823fd21d_4b0e_466c_aafa_f7fec0d68e17.slice/crio-5e063e1af264bd253227de3874adac6576dabed7c4fb84c6a772b5c807b7b905 WatchSource:0}: Error finding container 5e063e1af264bd253227de3874adac6576dabed7c4fb84c6a772b5c807b7b905: Status 404 returned error can't find the container with id 5e063e1af264bd253227de3874adac6576dabed7c4fb84c6a772b5c807b7b905
Nov 03 22:00:01.018059 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:01.018030    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ec7f5b_3d9d_464c_af0b_bb9a41811675.slice/crio-978331d89bb74532ef059772374ff9a3b3c46e065c68a3a4afac6c15bb690364 WatchSource:0}: Error finding container 978331d89bb74532ef059772374ff9a3b3c46e065c68a3a4afac6c15bb690364: Status 404 returned error can't find the container with id 978331d89bb74532ef059772374ff9a3b3c46e065c68a3a4afac6c15bb690364
Nov 03 22:00:01.019799 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.019777    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"]
Nov 03 22:00:01.046492 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.046466    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"]
Nov 03 22:00:01.135703 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:01.135172    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98452d71_ec0a_4c9c_9cbf_149b14027a0a.slice/crio-5fde0ee946e4bc5c6684cff3dbd54f0ad92aeffa85702b2ec385ce222b507e64 WatchSource:0}: Error finding container 5fde0ee946e4bc5c6684cff3dbd54f0ad92aeffa85702b2ec385ce222b507e64: Status 404 returned error can't find the container with id 5fde0ee946e4bc5c6684cff3dbd54f0ad92aeffa85702b2ec385ce222b507e64
Nov 03 22:00:01.462492 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.462462    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" event={"ID":"823fd21d-4b0e-466c-aafa-f7fec0d68e17","Type":"ContainerStarted","Data":"5e063e1af264bd253227de3874adac6576dabed7c4fb84c6a772b5c807b7b905"}
Nov 03 22:00:01.464465 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.464438    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" event={"ID":"d0515020-1ea8-4a15-81b3-381ec44b1532","Type":"ContainerStarted","Data":"5428427f6c0cb72a6d6888ff93405b3f2707e68bdaad550283a1a5a69d95d577"}
Nov 03 22:00:01.466225 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.466202    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerStarted","Data":"7f1a1d125d91b20f1334eda8b6772f2a6ef8ce796f4ad9e54a56347a02d8c6dd"}
Nov 03 22:00:01.466359 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.466251    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerStarted","Data":"ebf2252a15b8326730f309db872a5b0c0edaeb37eedffb7b947a8fc8cda88b2c"}
Nov 03 22:00:01.472860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.472814    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerStarted","Data":"5fde0ee946e4bc5c6684cff3dbd54f0ad92aeffa85702b2ec385ce222b507e64"}
Nov 03 22:00:01.474163 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.474141    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerStarted","Data":"978331d89bb74532ef059772374ff9a3b3c46e065c68a3a4afac6c15bb690364"}
Nov 03 22:00:01.475589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.475567    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerStarted","Data":"d23ec3a7864671a92cf8f77f180b954adf01e9218d1cb67212dc3fd4d5c6b3cd"}
Nov 03 22:00:01.475688 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.475602    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerStarted","Data":"e6b1e8663a62b7cba46fccdcd9fec0b0b4582d7fcee1cadfc353e98340c2ca3a"}
Nov 03 22:00:01.477058 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.477037    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" event={"ID":"16a630df-7f51-49d7-87d0-d13dd917b535","Type":"ContainerStarted","Data":"d7bad2d8cd59afcfb56f2d60f744038b136893d94382df99230e2453e1715f68"}
Nov 03 22:00:01.577570 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.577515    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" podStartSLOduration=1.577497983 podStartE2EDuration="1.577497983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.577358692 +0000 UTC m=+2672.561355244" watchObservedRunningTime="2026-11-03 22:00:01.577497983 +0000 UTC m=+2672.561494542"
Nov 03 22:00:01.689784 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.689726    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" podStartSLOduration=1.6897080500000001 podStartE2EDuration="1.68970805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.68883223 +0000 UTC m=+2672.672828810" watchObservedRunningTime="2026-11-03 22:00:01.68970805 +0000 UTC m=+2672.673704608"
Nov 03 22:00:01.800678 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.800561    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" podStartSLOduration=1.80054855 podStartE2EDuration="1.80054855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.800437812 +0000 UTC m=+2672.784434372" watchObservedRunningTime="2026-11-03 22:00:01.80054855 +0000 UTC m=+2672.784545091"
Nov 03 22:00:01.914041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:01.913977    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" podStartSLOduration=1.913958243 podStartE2EDuration="1.913958243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:01.91169731 +0000 UTC m=+2672.895693900" watchObservedRunningTime="2026-11-03 22:00:01.913958243 +0000 UTC m=+2672.897954803"
Nov 03 22:00:02.485542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.485511    5244 generic.go:356] "Generic (PLEG): container finished" podID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerID="c38ee1ea5e3ed6812f7b96aeae8190507ccc12a8d0b7ca8144f43a3812c9f5d5" exitCode=60
Nov 03 22:00:02.486001 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.485584    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerDied","Data":"c38ee1ea5e3ed6812f7b96aeae8190507ccc12a8d0b7ca8144f43a3812c9f5d5"}
Nov 03 22:00:02.486001 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.485844    5244 scope.go:122] "RemoveContainer" containerID="c38ee1ea5e3ed6812f7b96aeae8190507ccc12a8d0b7ca8144f43a3812c9f5d5"
Nov 03 22:00:02.487275 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.487246    5244 generic.go:356] "Generic (PLEG): container finished" podID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerID="d23ec3a7864671a92cf8f77f180b954adf01e9218d1cb67212dc3fd4d5c6b3cd" exitCode=60
Nov 03 22:00:02.487409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.487314    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerDied","Data":"d23ec3a7864671a92cf8f77f180b954adf01e9218d1cb67212dc3fd4d5c6b3cd"}
Nov 03 22:00:02.487627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.487611    5244 scope.go:122] "RemoveContainer" containerID="d23ec3a7864671a92cf8f77f180b954adf01e9218d1cb67212dc3fd4d5c6b3cd"
Nov 03 22:00:02.489106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.488990    5244 generic.go:356] "Generic (PLEG): container finished" podID="16a630df-7f51-49d7-87d0-d13dd917b535" containerID="ff5c3eb2cb9464fd9fb2c687b27d12330f030da699215902a1214463eabc6b33" exitCode=60
Nov 03 22:00:02.489106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.489071    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" event={"ID":"16a630df-7f51-49d7-87d0-d13dd917b535","Type":"ContainerDied","Data":"ff5c3eb2cb9464fd9fb2c687b27d12330f030da699215902a1214463eabc6b33"}
Nov 03 22:00:02.489293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.489277    5244 scope.go:122] "RemoveContainer" containerID="ff5c3eb2cb9464fd9fb2c687b27d12330f030da699215902a1214463eabc6b33"
Nov 03 22:00:02.490904 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.490879    5244 generic.go:356] "Generic (PLEG): container finished" podID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerID="63d1478e10fc096f37ac532b44ff901c393956992d1d86d64049e20417e6b79e" exitCode=60
Nov 03 22:00:02.490976 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.490949    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" event={"ID":"823fd21d-4b0e-466c-aafa-f7fec0d68e17","Type":"ContainerDied","Data":"63d1478e10fc096f37ac532b44ff901c393956992d1d86d64049e20417e6b79e"}
Nov 03 22:00:02.491143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.491118    5244 scope.go:122] "RemoveContainer" containerID="63d1478e10fc096f37ac532b44ff901c393956992d1d86d64049e20417e6b79e"
Nov 03 22:00:02.492613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.492590    5244 generic.go:356] "Generic (PLEG): container finished" podID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerID="d3da8fbf11f19d8cb1064bebeee27fb67c616d7a7b6e044c87cecf422ebc10fc" exitCode=60
Nov 03 22:00:02.492687 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.492668    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" event={"ID":"d0515020-1ea8-4a15-81b3-381ec44b1532","Type":"ContainerDied","Data":"d3da8fbf11f19d8cb1064bebeee27fb67c616d7a7b6e044c87cecf422ebc10fc"}
Nov 03 22:00:02.492822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.492809    5244 scope.go:122] "RemoveContainer" containerID="d3da8fbf11f19d8cb1064bebeee27fb67c616d7a7b6e044c87cecf422ebc10fc"
Nov 03 22:00:02.494150 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.494129    5244 generic.go:356] "Generic (PLEG): container finished" podID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerID="7f1a1d125d91b20f1334eda8b6772f2a6ef8ce796f4ad9e54a56347a02d8c6dd" exitCode=60
Nov 03 22:00:02.494270 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.494223    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerDied","Data":"7f1a1d125d91b20f1334eda8b6772f2a6ef8ce796f4ad9e54a56347a02d8c6dd"}
Nov 03 22:00:02.494409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.494395    5244 scope.go:122] "RemoveContainer" containerID="7f1a1d125d91b20f1334eda8b6772f2a6ef8ce796f4ad9e54a56347a02d8c6dd"
Nov 03 22:00:02.496555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.496536    5244 generic.go:356] "Generic (PLEG): container finished" podID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerID="d18e36d5b0498fdee058df7314b0df0cda3c3c509dfe50e2ea63618c49442e6f" exitCode=60
Nov 03 22:00:02.496655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.496587    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerDied","Data":"d18e36d5b0498fdee058df7314b0df0cda3c3c509dfe50e2ea63618c49442e6f"}
Nov 03 22:00:02.496760 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:02.496749    5244 scope.go:122] "RemoveContainer" containerID="d18e36d5b0498fdee058df7314b0df0cda3c3c509dfe50e2ea63618c49442e6f"
Nov 03 22:00:03.505123 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.505064    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerStarted","Data":"8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d"}
Nov 03 22:00:03.507066 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.507041    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerStarted","Data":"6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312"}
Nov 03 22:00:03.508859 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.508831    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerStarted","Data":"e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b"}
Nov 03 22:00:03.510887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.510861    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" event={"ID":"16a630df-7f51-49d7-87d0-d13dd917b535","Type":"ContainerStarted","Data":"1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3"}
Nov 03 22:00:03.512885 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.512859    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" event={"ID":"823fd21d-4b0e-466c-aafa-f7fec0d68e17","Type":"ContainerStarted","Data":"f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639"}
Nov 03 22:00:03.514834 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.514808    5244 generic.go:356] "Generic (PLEG): container finished" podID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerID="cdd52552dd24e0d22e8f873dee4767a615e5c40dcbd73f75809d62a5c5585994" exitCode=60
Nov 03 22:00:03.514946 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.514868    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" event={"ID":"d0515020-1ea8-4a15-81b3-381ec44b1532","Type":"ContainerDied","Data":"cdd52552dd24e0d22e8f873dee4767a615e5c40dcbd73f75809d62a5c5585994"}
Nov 03 22:00:03.514946 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.514932    5244 scope.go:122] "RemoveContainer" containerID="d3da8fbf11f19d8cb1064bebeee27fb67c616d7a7b6e044c87cecf422ebc10fc"
Nov 03 22:00:03.515130 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.515110    5244 scope.go:122] "RemoveContainer" containerID="cdd52552dd24e0d22e8f873dee4767a615e5c40dcbd73f75809d62a5c5585994"
Nov 03 22:00:03.515397 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:03.515352    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p_odh-ai-gateway-infra(d0515020-1ea8-4a15-81b3-381ec44b1532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532"
Nov 03 22:00:03.516872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.516821    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerStarted","Data":"6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827"}
Nov 03 22:00:03.616754 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:03.616698    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" podStartSLOduration=3.61668409 podStartE2EDuration="3.61668409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:03.616381769 +0000 UTC m=+2674.600378326" watchObservedRunningTime="2026-11-03 22:00:03.61668409 +0000 UTC m=+2674.600680648"
Nov 03 22:00:04.310668 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.310605    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" podStartSLOduration=4.310591259 podStartE2EDuration="4.310591259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:04.308629868 +0000 UTC m=+2675.292626452" watchObservedRunningTime="2026-11-03 22:00:04.310591259 +0000 UTC m=+2675.294587853"
Nov 03 22:00:04.526836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.526805    5244 scope.go:122] "RemoveContainer" containerID="cdd52552dd24e0d22e8f873dee4767a615e5c40dcbd73f75809d62a5c5585994"
Nov 03 22:00:04.527406 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.527149    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p_odh-ai-gateway-infra(d0515020-1ea8-4a15-81b3-381ec44b1532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532"
Nov 03 22:00:04.529360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.529045    5244 generic.go:356] "Generic (PLEG): container finished" podID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerID="6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827" exitCode=60
Nov 03 22:00:04.529360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.529094    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerDied","Data":"6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827"}
Nov 03 22:00:04.529360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.529165    5244 scope.go:122] "RemoveContainer" containerID="7f1a1d125d91b20f1334eda8b6772f2a6ef8ce796f4ad9e54a56347a02d8c6dd"
Nov 03 22:00:04.529950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.529928    5244 scope.go:122] "RemoveContainer" containerID="6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827"
Nov 03 22:00:04.530274 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.530250    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp_odh-ai-gateway-infra(e8a0cea9-bbb2-41b6-9c31-4e66f825f049)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049"
Nov 03 22:00:04.532613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.532589    5244 generic.go:356] "Generic (PLEG): container finished" podID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerID="8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d" exitCode=60
Nov 03 22:00:04.532713 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.532677    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerDied","Data":"8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d"}
Nov 03 22:00:04.532974 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.532958    5244 scope.go:122] "RemoveContainer" containerID="8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d"
Nov 03 22:00:04.533249 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.533229    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z_odh-ai-gateway-infra(98452d71-ec0a-4c9c-9cbf-149b14027a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a"
Nov 03 22:00:04.534947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.534924    5244 generic.go:356] "Generic (PLEG): container finished" podID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerID="6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312" exitCode=60
Nov 03 22:00:04.535267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.535026    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerDied","Data":"6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312"}
Nov 03 22:00:04.535414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.535397    5244 scope.go:122] "RemoveContainer" containerID="6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312"
Nov 03 22:00:04.535710 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.535683    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww_odh-ai-gateway-infra(73ec7f5b-3d9d-464c-af0b-bb9a41811675)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675"
Nov 03 22:00:04.537905 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.537878    5244 generic.go:356] "Generic (PLEG): container finished" podID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerID="e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b" exitCode=60
Nov 03 22:00:04.538008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.537974    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerDied","Data":"e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b"}
Nov 03 22:00:04.538225 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.538207    5244 scope.go:122] "RemoveContainer" containerID="e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b"
Nov 03 22:00:04.538563 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.538539    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj_odh-ai-gateway-infra(2bc6b000-1c16-43dd-b67f-c24903c56567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567"
Nov 03 22:00:04.543370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.540778    5244 generic.go:356] "Generic (PLEG): container finished" podID="16a630df-7f51-49d7-87d0-d13dd917b535" containerID="1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3" exitCode=60
Nov 03 22:00:04.543370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.540853    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" event={"ID":"16a630df-7f51-49d7-87d0-d13dd917b535","Type":"ContainerDied","Data":"1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3"}
Nov 03 22:00:04.543370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.541212    5244 scope.go:122] "RemoveContainer" containerID="1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3"
Nov 03 22:00:04.543370 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.541552    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm_odh-ai-gateway-infra(16a630df-7f51-49d7-87d0-d13dd917b535)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" podUID="16a630df-7f51-49d7-87d0-d13dd917b535"
Nov 03 22:00:04.543370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.543154    5244 generic.go:356] "Generic (PLEG): container finished" podID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerID="f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639" exitCode=60
Nov 03 22:00:04.543370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.543234    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" event={"ID":"823fd21d-4b0e-466c-aafa-f7fec0d68e17","Type":"ContainerDied","Data":"f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639"}
Nov 03 22:00:04.543764 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.543520    5244 scope.go:122] "RemoveContainer" containerID="f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639"
Nov 03 22:00:04.543817 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:04.543778    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-d6dbt_odh-ai-gateway-infra(823fd21d-4b0e-466c-aafa-f7fec0d68e17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17"
Nov 03 22:00:04.545940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.545920    5244 scope.go:122] "RemoveContainer" containerID="d18e36d5b0498fdee058df7314b0df0cda3c3c509dfe50e2ea63618c49442e6f"
Nov 03 22:00:04.562657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.562631    5244 scope.go:122] "RemoveContainer" containerID="c38ee1ea5e3ed6812f7b96aeae8190507ccc12a8d0b7ca8144f43a3812c9f5d5"
Nov 03 22:00:04.581320 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.581289    5244 scope.go:122] "RemoveContainer" containerID="d23ec3a7864671a92cf8f77f180b954adf01e9218d1cb67212dc3fd4d5c6b3cd"
Nov 03 22:00:04.597642 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.597531    5244 scope.go:122] "RemoveContainer" containerID="ff5c3eb2cb9464fd9fb2c687b27d12330f030da699215902a1214463eabc6b33"
Nov 03 22:00:04.662369 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:04.660918    5244 scope.go:122] "RemoveContainer" containerID="63d1478e10fc096f37ac532b44ff901c393956992d1d86d64049e20417e6b79e"
Nov 03 22:00:05.554954 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:05.554923    5244 scope.go:122] "RemoveContainer" containerID="6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312"
Nov 03 22:00:05.555490 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:05.555220    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww_odh-ai-gateway-infra(73ec7f5b-3d9d-464c-af0b-bb9a41811675)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675"
Nov 03 22:00:05.556630 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:05.556608    5244 scope.go:122] "RemoveContainer" containerID="e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b"
Nov 03 22:00:05.556916 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:05.556874    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj_odh-ai-gateway-infra(2bc6b000-1c16-43dd-b67f-c24903c56567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567"
Nov 03 22:00:05.558530 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:05.558498    5244 scope.go:122] "RemoveContainer" containerID="1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3"
Nov 03 22:00:05.558775 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:05.558746    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm_odh-ai-gateway-infra(16a630df-7f51-49d7-87d0-d13dd917b535)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" podUID="16a630df-7f51-49d7-87d0-d13dd917b535"
Nov 03 22:00:05.560411 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:05.560390    5244 scope.go:122] "RemoveContainer" containerID="f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639"
Nov 03 22:00:05.560653 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:05.560631    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-d6dbt_odh-ai-gateway-infra(823fd21d-4b0e-466c-aafa-f7fec0d68e17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17"
Nov 03 22:00:05.561863 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:05.561847    5244 scope.go:122] "RemoveContainer" containerID="6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827"
Nov 03 22:00:05.562048 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:05.562030    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp_odh-ai-gateway-infra(e8a0cea9-bbb2-41b6-9c31-4e66f825f049)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049"
Nov 03 22:00:05.563629 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:05.563609    5244 scope.go:122] "RemoveContainer" containerID="8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d"
Nov 03 22:00:05.563836 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:05.563815    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z_odh-ai-gateway-infra(98452d71-ec0a-4c9c-9cbf-149b14027a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a"
Nov 03 22:00:15.130012 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.129974    5244 scope.go:122] "RemoveContainer" containerID="cdd52552dd24e0d22e8f873dee4767a615e5c40dcbd73f75809d62a5c5585994"
Nov 03 22:00:15.130484 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.130210    5244 scope.go:122] "RemoveContainer" containerID="1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3"
Nov 03 22:00:15.130548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.130505    5244 scope.go:122] "RemoveContainer" containerID="8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d"
Nov 03 22:00:15.130638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.130618    5244 scope.go:122] "RemoveContainer" containerID="f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639"
Nov 03 22:00:15.626485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.626450    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerStarted","Data":"4efeaf4f396fff6accbb21551c0ecd2f2c601dace56c0274e3dd1a7b34e75e72"}
Nov 03 22:00:15.628448 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.628423    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" event={"ID":"d0515020-1ea8-4a15-81b3-381ec44b1532","Type":"ContainerStarted","Data":"a7bc2b749fe6707029de5f53805679e98d1d1da0a4b30cd79194041273e1e826"}
Nov 03 22:00:15.930633 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:15.930505    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" podStartSLOduration=15.930485136 podStartE2EDuration="15.930485136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:15.929999116 +0000 UTC m=+2686.913995676" watchObservedRunningTime="2026-11-03 22:00:15.930485136 +0000 UTC m=+2686.914481709"
Nov 03 22:00:16.130310 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.130267    5244 scope.go:122] "RemoveContainer" containerID="6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312"
Nov 03 22:00:16.130779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.130553    5244 scope.go:122] "RemoveContainer" containerID="e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b"
Nov 03 22:00:16.637896 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.637855    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerStarted","Data":"af3c35f644cadd55dce9dc8a9cf4fa1e42a3886a0a3a95f721f22597e8effcb8"}
Nov 03 22:00:16.640456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.640426    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerStarted","Data":"500ae80875795e906e752faf7749c02e5977b25d311d45e95ab536eb98fba309"}
Nov 03 22:00:16.642464 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.642441    5244 generic.go:356] "Generic (PLEG): container finished" podID="16a630df-7f51-49d7-87d0-d13dd917b535" containerID="309034bc1c5cb6067031679fa3ac0e770e791c7cbcb57b2924fab93e4d73943a" exitCode=60
Nov 03 22:00:16.642606 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.642466    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" event={"ID":"16a630df-7f51-49d7-87d0-d13dd917b535","Type":"ContainerDied","Data":"309034bc1c5cb6067031679fa3ac0e770e791c7cbcb57b2924fab93e4d73943a"}
Nov 03 22:00:16.642606 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.642525    5244 scope.go:122] "RemoveContainer" containerID="1eb27c66bd96e4ab9d5a284e61eea04ada43350b0a7f7c84e37355a42ae109d3"
Nov 03 22:00:16.642834 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.642810    5244 scope.go:122] "RemoveContainer" containerID="309034bc1c5cb6067031679fa3ac0e770e791c7cbcb57b2924fab93e4d73943a"
Nov 03 22:00:16.643073 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:16.643046    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm_odh-ai-gateway-infra(16a630df-7f51-49d7-87d0-d13dd917b535)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" podUID="16a630df-7f51-49d7-87d0-d13dd917b535"
Nov 03 22:00:16.644765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.644745    5244 generic.go:356] "Generic (PLEG): container finished" podID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerID="61fb268790e9ed048584b7c33bb48ed8246133b81d5f7b29b7bf4b794d79c8e0" exitCode=60
Nov 03 22:00:16.644849 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.644770    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" event={"ID":"823fd21d-4b0e-466c-aafa-f7fec0d68e17","Type":"ContainerDied","Data":"61fb268790e9ed048584b7c33bb48ed8246133b81d5f7b29b7bf4b794d79c8e0"}
Nov 03 22:00:16.645086 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.645067    5244 scope.go:122] "RemoveContainer" containerID="61fb268790e9ed048584b7c33bb48ed8246133b81d5f7b29b7bf4b794d79c8e0"
Nov 03 22:00:16.645311 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:16.645286    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895720-d6dbt_odh-ai-gateway-infra(823fd21d-4b0e-466c-aafa-f7fec0d68e17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17"
Nov 03 22:00:16.646765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.646746    5244 generic.go:356] "Generic (PLEG): container finished" podID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerID="a7bc2b749fe6707029de5f53805679e98d1d1da0a4b30cd79194041273e1e826" exitCode=60
Nov 03 22:00:16.646861 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.646815    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" event={"ID":"d0515020-1ea8-4a15-81b3-381ec44b1532","Type":"ContainerDied","Data":"a7bc2b749fe6707029de5f53805679e98d1d1da0a4b30cd79194041273e1e826"}
Nov 03 22:00:16.647121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.647103    5244 scope.go:122] "RemoveContainer" containerID="a7bc2b749fe6707029de5f53805679e98d1d1da0a4b30cd79194041273e1e826"
Nov 03 22:00:16.647309 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:16.647291    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p_odh-ai-gateway-infra(d0515020-1ea8-4a15-81b3-381ec44b1532)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532"
Nov 03 22:00:16.649176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.649155    5244 generic.go:356] "Generic (PLEG): container finished" podID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerID="4efeaf4f396fff6accbb21551c0ecd2f2c601dace56c0274e3dd1a7b34e75e72" exitCode=60
Nov 03 22:00:16.649268 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.649185    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerDied","Data":"4efeaf4f396fff6accbb21551c0ecd2f2c601dace56c0274e3dd1a7b34e75e72"}
Nov 03 22:00:16.649525 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.649511    5244 scope.go:122] "RemoveContainer" containerID="4efeaf4f396fff6accbb21551c0ecd2f2c601dace56c0274e3dd1a7b34e75e72"
Nov 03 22:00:16.649746 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:16.649727    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z_odh-ai-gateway-infra(98452d71-ec0a-4c9c-9cbf-149b14027a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a"
Nov 03 22:00:16.656198 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.656178    5244 scope.go:122] "RemoveContainer" containerID="f5ae6e015116d3a5a7dc66861d9b08638fcb1d65cce33e714dd33ba55b256639"
Nov 03 22:00:16.674869 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.674842    5244 scope.go:122] "RemoveContainer" containerID="cdd52552dd24e0d22e8f873dee4767a615e5c40dcbd73f75809d62a5c5585994"
Nov 03 22:00:16.695046 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.695023    5244 scope.go:122] "RemoveContainer" containerID="8c26d49420ccd61ab31c1179daf57660e433e5f58754d7359027bbb8da419e1d"
Nov 03 22:00:16.748274 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.748236    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"]
Nov 03 22:00:16.940137 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:16.940102    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"]
Nov 03 22:00:17.656503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.656422    5244 generic.go:356] "Generic (PLEG): container finished" podID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerID="af3c35f644cadd55dce9dc8a9cf4fa1e42a3886a0a3a95f721f22597e8effcb8" exitCode=60
Nov 03 22:00:17.656503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.656487    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerDied","Data":"af3c35f644cadd55dce9dc8a9cf4fa1e42a3886a0a3a95f721f22597e8effcb8"}
Nov 03 22:00:17.657025 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.656544    5244 scope.go:122] "RemoveContainer" containerID="6373f308e50d4fcbab42a687714e55c0fdeb4ef480546f2b6fea315dfb384312"
Nov 03 22:00:17.657025 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.656956    5244 scope.go:122] "RemoveContainer" containerID="af3c35f644cadd55dce9dc8a9cf4fa1e42a3886a0a3a95f721f22597e8effcb8"
Nov 03 22:00:17.657298 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:17.657278    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww_odh-ai-gateway-infra(73ec7f5b-3d9d-464c-af0b-bb9a41811675)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675"
Nov 03 22:00:17.658767 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.658746    5244 generic.go:356] "Generic (PLEG): container finished" podID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerID="500ae80875795e906e752faf7749c02e5977b25d311d45e95ab536eb98fba309" exitCode=60
Nov 03 22:00:17.658882 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.658823    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerDied","Data":"500ae80875795e906e752faf7749c02e5977b25d311d45e95ab536eb98fba309"}
Nov 03 22:00:17.659574 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.659207    5244 scope.go:122] "RemoveContainer" containerID="500ae80875795e906e752faf7749c02e5977b25d311d45e95ab536eb98fba309"
Nov 03 22:00:17.659574 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:17.659530    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj_odh-ai-gateway-infra(2bc6b000-1c16-43dd-b67f-c24903c56567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567"
Nov 03 22:00:17.668776 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.668757    5244 scope.go:122] "RemoveContainer" containerID="e9a2c999e0b3bad15eb7351b666aac225ff6fe6a9a19538232d50567f374ff7b"
Nov 03 22:00:17.779648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.779609    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"]
Nov 03 22:00:17.813892 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.813861    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:17.849915 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.849890    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:17.889518 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.889487    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"]
Nov 03 22:00:17.893179 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.893155    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca\") pod \"d0515020-1ea8-4a15-81b3-381ec44b1532\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") "
Nov 03 22:00:17.893388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.893239    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx\" (UniqueName: \"kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx\") pod \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") "
Nov 03 22:00:17.893388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.893299    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t\" (UniqueName: \"kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t\") pod \"d0515020-1ea8-4a15-81b3-381ec44b1532\" (UID: \"d0515020-1ea8-4a15-81b3-381ec44b1532\") "
Nov 03 22:00:17.893388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.893364    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca\") pod \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\" (UID: \"98452d71-ec0a-4c9c-9cbf-149b14027a0a\") "
Nov 03 22:00:17.893634 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.893610    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca" pod "d0515020-1ea8-4a15-81b3-381ec44b1532" (UID: "d0515020-1ea8-4a15-81b3-381ec44b1532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:17.893818 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.893798    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca" pod "98452d71-ec0a-4c9c-9cbf-149b14027a0a" (UID: "98452d71-ec0a-4c9c-9cbf-149b14027a0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:17.895563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.895538    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t" pod "d0515020-1ea8-4a15-81b3-381ec44b1532" (UID: "d0515020-1ea8-4a15-81b3-381ec44b1532"). InnerVolumeSpecName "kube-api-access-vds8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:17.895674 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.895651    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx" pod "98452d71-ec0a-4c9c-9cbf-149b14027a0a" (UID: "98452d71-ec0a-4c9c-9cbf-149b14027a0a"). InnerVolumeSpecName "kube-api-access-jkslx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:17.994311 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.994278    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98452d71-ec0a-4c9c-9cbf-149b14027a0a-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:17.994311 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.994305    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0515020-1ea8-4a15-81b3-381ec44b1532-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:17.994311 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.994317    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkslx\" (UniqueName: \"kubernetes.io/projected/98452d71-ec0a-4c9c-9cbf-149b14027a0a-kube-api-access-jkslx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:17.994589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:17.994350    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vds8t\" (UniqueName: \"kubernetes.io/projected/d0515020-1ea8-4a15-81b3-381ec44b1532-kube-api-access-vds8t\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.130004 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.129972    5244 scope.go:122] "RemoveContainer" containerID="6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827"
Nov 03 22:00:18.150665 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.150637    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"]
Nov 03 22:00:18.358013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.357984    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:18.374262 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.374211    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"]
Nov 03 22:00:18.398567 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.398525    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca\") pod \"16a630df-7f51-49d7-87d0-d13dd917b535\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") "
Nov 03 22:00:18.398746 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.398718    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t\" (UniqueName: \"kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t\") pod \"16a630df-7f51-49d7-87d0-d13dd917b535\" (UID: \"16a630df-7f51-49d7-87d0-d13dd917b535\") "
Nov 03 22:00:18.399034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.398992    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca" pod "16a630df-7f51-49d7-87d0-d13dd917b535" (UID: "16a630df-7f51-49d7-87d0-d13dd917b535"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:18.399222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.399202    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16a630df-7f51-49d7-87d0-d13dd917b535-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.402847 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.402811    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t" pod "16a630df-7f51-49d7-87d0-d13dd917b535" (UID: "16a630df-7f51-49d7-87d0-d13dd917b535"). InnerVolumeSpecName "kube-api-access-bpt9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:18.500498 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.500463    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpt9t\" (UniqueName: \"kubernetes.io/projected/16a630df-7f51-49d7-87d0-d13dd917b535-kube-api-access-bpt9t\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.535205 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.535178    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:18.601299 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.601211    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca\") pod \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") "
Nov 03 22:00:18.601299 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.601262    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8\" (UniqueName: \"kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8\") pod \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\" (UID: \"823fd21d-4b0e-466c-aafa-f7fec0d68e17\") "
Nov 03 22:00:18.601615 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.601572    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca" pod "823fd21d-4b0e-466c-aafa-f7fec0d68e17" (UID: "823fd21d-4b0e-466c-aafa-f7fec0d68e17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:18.601706 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.601673    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/823fd21d-4b0e-466c-aafa-f7fec0d68e17-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.603710 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.603685    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8" pod "823fd21d-4b0e-466c-aafa-f7fec0d68e17" (UID: "823fd21d-4b0e-466c-aafa-f7fec0d68e17"). InnerVolumeSpecName "kube-api-access-znlv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:18.673674 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.673639    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"
Nov 03 22:00:18.674143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.673638    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p" event={"ID":"d0515020-1ea8-4a15-81b3-381ec44b1532","Type":"ContainerDied","Data":"5428427f6c0cb72a6d6888ff93405b3f2707e68bdaad550283a1a5a69d95d577"}
Nov 03 22:00:18.674143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.673815    5244 scope.go:122] "RemoveContainer" containerID="a7bc2b749fe6707029de5f53805679e98d1d1da0a4b30cd79194041273e1e826"
Nov 03 22:00:18.676360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.676318    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerStarted","Data":"5d04ab32356e61d38ab2858e4e781b8a7992e728c47f270f9e10114d8cf24faf"}
Nov 03 22:00:18.677949 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.677933    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"
Nov 03 22:00:18.678059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.677967    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z" event={"ID":"98452d71-ec0a-4c9c-9cbf-149b14027a0a","Type":"ContainerDied","Data":"5fde0ee946e4bc5c6684cff3dbd54f0ad92aeffa85702b2ec385ce222b507e64"}
Nov 03 22:00:18.692887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.692723    5244 scope.go:122] "RemoveContainer" containerID="4efeaf4f396fff6accbb21551c0ecd2f2c601dace56c0274e3dd1a7b34e75e72"
Nov 03 22:00:18.695355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.694920    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm" event={"ID":"16a630df-7f51-49d7-87d0-d13dd917b535","Type":"ContainerDied","Data":"d7bad2d8cd59afcfb56f2d60f744038b136893d94382df99230e2453e1715f68"}
Nov 03 22:00:18.695355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.695035    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"
Nov 03 22:00:18.703230 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.703196    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlv8\" (UniqueName: \"kubernetes.io/projected/823fd21d-4b0e-466c-aafa-f7fec0d68e17-kube-api-access-znlv8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.704037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.703761    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt" event={"ID":"823fd21d-4b0e-466c-aafa-f7fec0d68e17","Type":"ContainerDied","Data":"5e063e1af264bd253227de3874adac6576dabed7c4fb84c6a772b5c807b7b905"}
Nov 03 22:00:18.704037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.703835    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"
Nov 03 22:00:18.707501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.707477    5244 scope.go:122] "RemoveContainer" containerID="309034bc1c5cb6067031679fa3ac0e770e791c7cbcb57b2924fab93e4d73943a"
Nov 03 22:00:18.726481 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.726448    5244 scope.go:122] "RemoveContainer" containerID="61fb268790e9ed048584b7c33bb48ed8246133b81d5f7b29b7bf4b794d79c8e0"
Nov 03 22:00:18.826312 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.826279    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"]
Nov 03 22:00:18.828498 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.828474    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-eb3520-29895720-rnp5z"]
Nov 03 22:00:18.842375 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.841503    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:18.852664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.852611    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:18.905375 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.905310    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca\") pod \"2bc6b000-1c16-43dd-b67f-c24903c56567\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") "
Nov 03 22:00:18.905548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.905439    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg\" (UniqueName: \"kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg\") pod \"2bc6b000-1c16-43dd-b67f-c24903c56567\" (UID: \"2bc6b000-1c16-43dd-b67f-c24903c56567\") "
Nov 03 22:00:18.905548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.905494    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca\") pod \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") "
Nov 03 22:00:18.905648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.905615    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r\" (UniqueName: \"kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r\") pod \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\" (UID: \"73ec7f5b-3d9d-464c-af0b-bb9a41811675\") "
Nov 03 22:00:18.906017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.905918    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca" pod "73ec7f5b-3d9d-464c-af0b-bb9a41811675" (UID: "73ec7f5b-3d9d-464c-af0b-bb9a41811675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:18.906017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.905991    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ec7f5b-3d9d-464c-af0b-bb9a41811675-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:18.906528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.906316    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca" pod "2bc6b000-1c16-43dd-b67f-c24903c56567" (UID: "2bc6b000-1c16-43dd-b67f-c24903c56567"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:18.907924 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.907895    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r" pod "73ec7f5b-3d9d-464c-af0b-bb9a41811675" (UID: "73ec7f5b-3d9d-464c-af0b-bb9a41811675"). InnerVolumeSpecName "kube-api-access-rlx4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:18.908106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:18.908087    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg" pod "2bc6b000-1c16-43dd-b67f-c24903c56567" (UID: "2bc6b000-1c16-43dd-b67f-c24903c56567"). InnerVolumeSpecName "kube-api-access-q9ftg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:19.007224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.007187    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc6b000-1c16-43dd-b67f-c24903c56567-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.007224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.007217    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ftg\" (UniqueName: \"kubernetes.io/projected/2bc6b000-1c16-43dd-b67f-c24903c56567-kube-api-access-q9ftg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.007224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.007228    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlx4r\" (UniqueName: \"kubernetes.io/projected/73ec7f5b-3d9d-464c-af0b-bb9a41811675-kube-api-access-rlx4r\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:19.069603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.069562    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"]
Nov 03 22:00:19.070000 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.069986    5244 cpu_manager.go:414] "containerMap: removing container" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070001    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070012    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070017    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070028    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070033    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070038    5244 cpu_manager.go:414] "containerMap: removing container" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:19.070040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070043    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070055    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070059    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070068    5244 cpu_manager.go:414] "containerMap: removing container" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070073    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070080    5244 cpu_manager.go:414] "containerMap: removing container" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070085    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070092    5244 cpu_manager.go:414] "containerMap: removing container" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070096    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070104    5244 cpu_manager.go:414] "containerMap: removing container" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070108    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070113    5244 cpu_manager.go:414] "containerMap: removing container" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070117    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070129    5244 cpu_manager.go:414] "containerMap: removing container" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070132    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070141    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070144    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070155    5244 cpu_manager.go:414] "containerMap: removing container" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070159    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070225    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070231    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070238    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070244    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070250    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070256    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070261    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070267    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.070267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070273    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070280    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070286    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070291    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070296    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070300    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070305    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" containerName="cleanup"
Nov 03 22:00:19.071049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.070310    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:19.073113 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.073090    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.108579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.108550    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2zk\" (UniqueName: \"kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.108721 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.108610    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.108721 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.108686    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.126548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.126521    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:00:19.128998 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.128964    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"]
Nov 03 22:00:19.139072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.139047    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" path="/var/lib/kubelet/pods/98452d71-ec0a-4c9c-9cbf-149b14027a0a/volumes"
Nov 03 22:00:19.209680 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.209647    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"]
Nov 03 22:00:19.209876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.209814    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2zk\" (UniqueName: \"kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.209937 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.209873    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.209937 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.209896    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.210704 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.210670    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.212827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.212795    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.213013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.212994    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895720-rz99p"]
Nov 03 22:00:19.315964 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.315934    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2zk\" (UniqueName: \"kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk\") pod \"maas-api-revoke-sub-758cff82-gfmht\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.369446 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.369366    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"]
Nov 03 22:00:19.372694 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.372666    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-95d604-29895720-8lkrm"]
Nov 03 22:00:19.409773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.409738    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:19.597066 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.597028    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"]
Nov 03 22:00:19.599029 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.599001    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895720-d6dbt"]
Nov 03 22:00:19.721662 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.721629    5244 generic.go:356] "Generic (PLEG): container finished" podID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerID="5d04ab32356e61d38ab2858e4e781b8a7992e728c47f270f9e10114d8cf24faf" exitCode=60
Nov 03 22:00:19.722158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.721709    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerDied","Data":"5d04ab32356e61d38ab2858e4e781b8a7992e728c47f270f9e10114d8cf24faf"}
Nov 03 22:00:19.722158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.721811    5244 scope.go:122] "RemoveContainer" containerID="6ecff4caebb25e59f4c91da7ae4bd424c26da10059be69ef08f66f9476b51827"
Nov 03 22:00:19.722291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.722167    5244 scope.go:122] "RemoveContainer" containerID="5d04ab32356e61d38ab2858e4e781b8a7992e728c47f270f9e10114d8cf24faf"
Nov 03 22:00:19.733753 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:19.722460    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp_odh-ai-gateway-infra(e8a0cea9-bbb2-41b6-9c31-4e66f825f049)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049"
Nov 03 22:00:19.733753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.724833    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"
Nov 03 22:00:19.733753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.724834    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww" event={"ID":"73ec7f5b-3d9d-464c-af0b-bb9a41811675","Type":"ContainerDied","Data":"978331d89bb74532ef059772374ff9a3b3c46e065c68a3a4afac6c15bb690364"}
Nov 03 22:00:19.733753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.726307    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj" event={"ID":"2bc6b000-1c16-43dd-b67f-c24903c56567","Type":"ContainerDied","Data":"e6b1e8663a62b7cba46fccdcd9fec0b0b4582d7fcee1cadfc353e98340c2ca3a"}
Nov 03 22:00:19.733753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.726315    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"
Nov 03 22:00:19.736537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.736513    5244 scope.go:122] "RemoveContainer" containerID="af3c35f644cadd55dce9dc8a9cf4fa1e42a3886a0a3a95f721f22597e8effcb8"
Nov 03 22:00:19.746064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.746040    5244 scope.go:122] "RemoveContainer" containerID="500ae80875795e906e752faf7749c02e5977b25d311d45e95ab536eb98fba309"
Nov 03 22:00:19.782657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:19.782634    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"]
Nov 03 22:00:19.785085 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:19.785053    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96415ea_7da2_4c38_9ff3_557bcbb71989.slice/crio-985adc0d747e96b09c861a52066775977e2b4d2b10d306e1c6c41cc90997eb47 WatchSource:0}: Error finding container 985adc0d747e96b09c861a52066775977e2b4d2b10d306e1c6c41cc90997eb47: Status 404 returned error can't find the container with id 985adc0d747e96b09c861a52066775977e2b4d2b10d306e1c6c41cc90997eb47
Nov 03 22:00:20.213102 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.213069    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"]
Nov 03 22:00:20.216575 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.216549    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895720-n4vww"]
Nov 03 22:00:20.374173 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.374087    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"]
Nov 03 22:00:20.377980 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.377945    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f463c9-29895720-pqdkj"]
Nov 03 22:00:20.493778 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.493744    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"]
Nov 03 22:00:20.742275 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.742248    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht" event={"ID":"b96415ea-7da2-4c38-9ff3-557bcbb71989","Type":"ContainerStarted","Data":"4373cbc9ad6edd5d751a7317b7556fdfd2fade9a1bf64e6c7e83b5c0a890c0f0"}
Nov 03 22:00:20.742714 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.742292    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht" event={"ID":"b96415ea-7da2-4c38-9ff3-557bcbb71989","Type":"ContainerStarted","Data":"985adc0d747e96b09c861a52066775977e2b4d2b10d306e1c6c41cc90997eb47"}
Nov 03 22:00:20.875614 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.875590    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:20.931596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.931562    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c\" (UniqueName: \"kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c\") pod \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") "
Nov 03 22:00:20.931775 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.931759    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca\") pod \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\" (UID: \"e8a0cea9-bbb2-41b6-9c31-4e66f825f049\") "
Nov 03 22:00:20.932067 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.932047    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca" pod "e8a0cea9-bbb2-41b6-9c31-4e66f825f049" (UID: "e8a0cea9-bbb2-41b6-9c31-4e66f825f049"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:20.933859 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:20.933837    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c" pod "e8a0cea9-bbb2-41b6-9c31-4e66f825f049" (UID: "e8a0cea9-bbb2-41b6-9c31-4e66f825f049"). InnerVolumeSpecName "kube-api-access-llr4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:21.033470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.033389    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llr4c\" (UniqueName: \"kubernetes.io/projected/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-kube-api-access-llr4c\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:21.033470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.033418    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a0cea9-bbb2-41b6-9c31-4e66f825f049-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:21.136357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.136301    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a630df-7f51-49d7-87d0-d13dd917b535" path="/var/lib/kubelet/pods/16a630df-7f51-49d7-87d0-d13dd917b535/volumes"
Nov 03 22:00:21.136744 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.136728    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" path="/var/lib/kubelet/pods/2bc6b000-1c16-43dd-b67f-c24903c56567/volumes"
Nov 03 22:00:21.137071 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.137059    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" path="/var/lib/kubelet/pods/73ec7f5b-3d9d-464c-af0b-bb9a41811675/volumes"
Nov 03 22:00:21.137757 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.137743    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" path="/var/lib/kubelet/pods/823fd21d-4b0e-466c-aafa-f7fec0d68e17/volumes"
Nov 03 22:00:21.138130 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.138114    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" path="/var/lib/kubelet/pods/d0515020-1ea8-4a15-81b3-381ec44b1532/volumes"
Nov 03 22:00:21.749477 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.749447    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"
Nov 03 22:00:21.749477 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.749465    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp" event={"ID":"e8a0cea9-bbb2-41b6-9c31-4e66f825f049","Type":"ContainerDied","Data":"ebf2252a15b8326730f309db872a5b0c0edaeb37eedffb7b947a8fc8cda88b2c"}
Nov 03 22:00:21.750141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.749524    5244 scope.go:122] "RemoveContainer" containerID="5d04ab32356e61d38ab2858e4e781b8a7992e728c47f270f9e10114d8cf24faf"
Nov 03 22:00:21.752627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.752602    5244 generic.go:356] "Generic (PLEG): container finished" podID="b96415ea-7da2-4c38-9ff3-557bcbb71989" containerID="4373cbc9ad6edd5d751a7317b7556fdfd2fade9a1bf64e6c7e83b5c0a890c0f0" exitCode=0
Nov 03 22:00:21.752749 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.752636    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht" event={"ID":"b96415ea-7da2-4c38-9ff3-557bcbb71989","Type":"ContainerDied","Data":"4373cbc9ad6edd5d751a7317b7556fdfd2fade9a1bf64e6c7e83b5c0a890c0f0"}
Nov 03 22:00:21.922408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.922380    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:21.961068 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.961023    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"]
Nov 03 22:00:21.968779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:21.968749    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895720-hpklp"]
Nov 03 22:00:22.052795 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.052716    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca\") pod \"b96415ea-7da2-4c38-9ff3-557bcbb71989\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") "
Nov 03 22:00:22.052958 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.052797    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk\" (UniqueName: \"kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk\") pod \"b96415ea-7da2-4c38-9ff3-557bcbb71989\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") "
Nov 03 22:00:22.052958 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.052906    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca\") pod \"b96415ea-7da2-4c38-9ff3-557bcbb71989\" (UID: \"b96415ea-7da2-4c38-9ff3-557bcbb71989\") "
Nov 03 22:00:22.053214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.053184    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca" pod "b96415ea-7da2-4c38-9ff3-557bcbb71989" (UID: "b96415ea-7da2-4c38-9ff3-557bcbb71989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:22.053393 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.053374    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:22.054941 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.054921    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca" pod "b96415ea-7da2-4c38-9ff3-557bcbb71989" (UID: "b96415ea-7da2-4c38-9ff3-557bcbb71989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:00:22.055040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.055023    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk" pod "b96415ea-7da2-4c38-9ff3-557bcbb71989" (UID: "b96415ea-7da2-4c38-9ff3-557bcbb71989"). InnerVolumeSpecName "kube-api-access-vt2zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:22.154214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.154176    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt2zk\" (UniqueName: \"kubernetes.io/projected/b96415ea-7da2-4c38-9ff3-557bcbb71989-kube-api-access-vt2zk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:22.154214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.154208    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96415ea-7da2-4c38-9ff3-557bcbb71989-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:22.436748 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:22.436702    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:00:22.759957 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.759880    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"
Nov 03 22:00:22.759957 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.759920    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht" event={"ID":"b96415ea-7da2-4c38-9ff3-557bcbb71989","Type":"ContainerDied","Data":"985adc0d747e96b09c861a52066775977e2b4d2b10d306e1c6c41cc90997eb47"}
Nov 03 22:00:22.760414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:22.759967    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="985adc0d747e96b09c861a52066775977e2b4d2b10d306e1c6c41cc90997eb47"
Nov 03 22:00:23.135554 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:23.135472    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" path="/var/lib/kubelet/pods/e8a0cea9-bbb2-41b6-9c31-4e66f825f049/volumes"
Nov 03 22:00:28.535498 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:28.535464    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:00:29.852080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:29.852041    5244 scope.go:122] "RemoveContainer" containerID="b8f16e35569f9180b9124c5d008d6c03ad355a9adff09e088e7d30919fb49495"
Nov 03 22:00:29.861407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:29.861384    5244 scope.go:122] "RemoveContainer" containerID="fa472909057bcb82a5632189679c74204179a2f52b41b55822045d0d29dd1a89"
Nov 03 22:00:36.502142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502055    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"]
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502503    5244 cpu_manager.go:414] "containerMap: removing container" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502516    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823fd21d-4b0e-466c-aafa-f7fec0d68e17" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502528    5244 cpu_manager.go:414] "containerMap: removing container" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502533    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98452d71-ec0a-4c9c-9cbf-149b14027a0a" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502542    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502548    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502557    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502562    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502572    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502576    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc6b000-1c16-43dd-b67f-c24903c56567" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502595    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b96415ea-7da2-4c38-9ff3-557bcbb71989" containerName="revoke-keys"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502599    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96415ea-7da2-4c38-9ff3-557bcbb71989" containerName="revoke-keys"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502606    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502611    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502622    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502626    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502631    5244 cpu_manager.go:414] "containerMap: removing container" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502635    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502700    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96415ea-7da2-4c38-9ff3-557bcbb71989" containerName="revoke-keys"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502709    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502715    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0515020-1ea8-4a15-81b3-381ec44b1532" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502725    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:36.504628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.502732    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ec7f5b-3d9d-464c-af0b-bb9a41811675" containerName="cleanup"
Nov 03 22:00:36.505835 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.505820    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.562407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.562370    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"]
Nov 03 22:00:36.608920 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.608885    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:00:36.609107 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.608967    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:00:36.623136 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.623099    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76cw\" (UniqueName: \"kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.623291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.623229    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.623291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.623261    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.724506 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.724466    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.724721 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.724516    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.724721 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.724587    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76cw\" (UniqueName: \"kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.725262 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.725236    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.726920 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.726893    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.780731 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.780654    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76cw\" (UniqueName: \"kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw\") pod \"maas-api-revoke-sub-58d7e957-hbxbc\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:36.819256 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:36.819222    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:37.193461 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:37.193431    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"]
Nov 03 22:00:37.195746 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:37.195708    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055b8496_b4ed_4c0b_9367_c95ba5d4b230.slice/crio-6fc4d89d9df948549bea86d2afee065119374e4d04083892db1a3b21841b596d WatchSource:0}: Error finding container 6fc4d89d9df948549bea86d2afee065119374e4d04083892db1a3b21841b596d: Status 404 returned error can't find the container with id 6fc4d89d9df948549bea86d2afee065119374e4d04083892db1a3b21841b596d
Nov 03 22:00:37.849083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:37.848995    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc" event={"ID":"055b8496-b4ed-4c0b-9367-c95ba5d4b230","Type":"ContainerStarted","Data":"88ee3a2d03f4eabd8a7fe952d04386564b754043b95c49d0f342b18e28b09d4d"}
Nov 03 22:00:37.849083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:37.849053    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc" event={"ID":"055b8496-b4ed-4c0b-9367-c95ba5d4b230","Type":"ContainerStarted","Data":"6fc4d89d9df948549bea86d2afee065119374e4d04083892db1a3b21841b596d"}
Nov 03 22:00:37.970632 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:37.970580    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc" podStartSLOduration=1.970564636 podStartE2EDuration="1.970564636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:37.968792189 +0000 UTC m=+2708.952788776" watchObservedRunningTime="2026-11-03 22:00:37.970564636 +0000 UTC m=+2708.954561203"
Nov 03 22:00:38.028958 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.028924    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"]
Nov 03 22:00:38.029464 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.029450    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a0cea9-bbb2-41b6-9c31-4e66f825f049" containerName="cleanup"
Nov 03 22:00:38.032503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.032481    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.044373 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.044345    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzrx\" (UniqueName: \"kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx\") pod \"payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.044506 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.044433    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.087709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.087677    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5bd35c-dockercfg-f8xpv\""
Nov 03 22:00:38.088022 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.087676    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5bd35c\""
Nov 03 22:00:38.090588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.090555    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"]
Nov 03 22:00:38.145790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.145749    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzrx\" (UniqueName: \"kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx\") pod \"payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.146120 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.146099    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.147467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.147444    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.189723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.189687    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"]
Nov 03 22:00:38.193848 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.193828    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.243044 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.243008    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzrx\" (UniqueName: \"kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx\") pod \"payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.247733 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.247700    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.247923 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.247819    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mxs5\" (UniqueName: \"kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5\") pod \"payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.251537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.251509    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"]
Nov 03 22:00:38.345744 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.345710    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.349301 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.348895    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.349301 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.348972    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mxs5\" (UniqueName: \"kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5\") pod \"payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.349603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.349580    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.406872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.406848    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mxs5\" (UniqueName: \"kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5\") pod \"payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.508840 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.508804    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:38.727011 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.726980    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"]
Nov 03 22:00:38.729669 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:38.729632    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa95d5b1_c1ca_4a60_9695_6f151886aeb8.slice/crio-c04d46b94771d6ccb709def5df038becf3394e778cc6a1da75216e56f92a5afb WatchSource:0}: Error finding container c04d46b94771d6ccb709def5df038becf3394e778cc6a1da75216e56f92a5afb: Status 404 returned error can't find the container with id c04d46b94771d6ccb709def5df038becf3394e778cc6a1da75216e56f92a5afb
Nov 03 22:00:38.856550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.856508    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6" event={"ID":"fa95d5b1-c1ca-4a60-9695-6f151886aeb8","Type":"ContainerStarted","Data":"324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6"}
Nov 03 22:00:38.857091 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.856581    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6" event={"ID":"fa95d5b1-c1ca-4a60-9695-6f151886aeb8","Type":"ContainerStarted","Data":"c04d46b94771d6ccb709def5df038becf3394e778cc6a1da75216e56f92a5afb"}
Nov 03 22:00:38.857091 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.856687    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:38.858264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.858237    5244 generic.go:356] "Generic (PLEG): container finished" podID="055b8496-b4ed-4c0b-9367-c95ba5d4b230" containerID="88ee3a2d03f4eabd8a7fe952d04386564b754043b95c49d0f342b18e28b09d4d" exitCode=0
Nov 03 22:00:38.858407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.858278    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc" event={"ID":"055b8496-b4ed-4c0b-9367-c95ba5d4b230","Type":"ContainerDied","Data":"88ee3a2d03f4eabd8a7fe952d04386564b754043b95c49d0f342b18e28b09d4d"}
Nov 03 22:00:38.890798 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.890764    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"]
Nov 03 22:00:38.896374 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:38.895040    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56564cc_8866_42de_9dc0_4923ee287e65.slice/crio-239602d7e85472654cba564e34542143c90a7d31219be22ff88e100a86788cfa WatchSource:0}: Error finding container 239602d7e85472654cba564e34542143c90a7d31219be22ff88e100a86788cfa: Status 404 returned error can't find the container with id 239602d7e85472654cba564e34542143c90a7d31219be22ff88e100a86788cfa
Nov 03 22:00:38.975367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:38.975087    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6" podStartSLOduration=1.975068033 podStartE2EDuration="1.975068033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:38.973516005 +0000 UTC m=+2709.957512564" watchObservedRunningTime="2026-11-03 22:00:38.975068033 +0000 UTC m=+2709.959064590"
Nov 03 22:00:39.865085 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:39.865046    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4" event={"ID":"e56564cc-8866-42de-9dc0-4923ee287e65","Type":"ContainerStarted","Data":"b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075"}
Nov 03 22:00:39.865648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:39.865099    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4" event={"ID":"e56564cc-8866-42de-9dc0-4923ee287e65","Type":"ContainerStarted","Data":"239602d7e85472654cba564e34542143c90a7d31219be22ff88e100a86788cfa"}
Nov 03 22:00:39.865648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:39.865534    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:39.979670 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:39.979618    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4" podStartSLOduration=1.979602112 podStartE2EDuration="1.979602112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:39.9776058 +0000 UTC m=+2710.961602384" watchObservedRunningTime="2026-11-03 22:00:39.979602112 +0000 UTC m=+2710.963598671"
Nov 03 22:00:40.013790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.013766    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:40.072284 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.072246    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw\" (UniqueName: \"kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw\") pod \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") "
Nov 03 22:00:40.072284 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.072289    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca\") pod \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") "
Nov 03 22:00:40.072583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.072319    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca\") pod \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\" (UID: \"055b8496-b4ed-4c0b-9367-c95ba5d4b230\") "
Nov 03 22:00:40.072742 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.072717    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca" pod "055b8496-b4ed-4c0b-9367-c95ba5d4b230" (UID: "055b8496-b4ed-4c0b-9367-c95ba5d4b230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:40.072839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.072826    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:40.074593 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.074571    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca" pod "055b8496-b4ed-4c0b-9367-c95ba5d4b230" (UID: "055b8496-b4ed-4c0b-9367-c95ba5d4b230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:00:40.074776 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.074760    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw" pod "055b8496-b4ed-4c0b-9367-c95ba5d4b230" (UID: "055b8496-b4ed-4c0b-9367-c95ba5d4b230"). InnerVolumeSpecName "kube-api-access-m76cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:40.174170 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.174074    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m76cw\" (UniqueName: \"kubernetes.io/projected/055b8496-b4ed-4c0b-9367-c95ba5d4b230-kube-api-access-m76cw\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:40.174170 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.174105    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/055b8496-b4ed-4c0b-9367-c95ba5d4b230-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:40.872782 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.872753    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"
Nov 03 22:00:40.873179 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.872829    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc" event={"ID":"055b8496-b4ed-4c0b-9367-c95ba5d4b230","Type":"ContainerDied","Data":"6fc4d89d9df948549bea86d2afee065119374e4d04083892db1a3b21841b596d"}
Nov 03 22:00:40.873179 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:40.872872    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fc4d89d9df948549bea86d2afee065119374e4d04083892db1a3b21841b596d"
Nov 03 22:00:41.379255 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.379212    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"]
Nov 03 22:00:41.379728 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.379713    5244 cpu_manager.go:414] "containerMap: removing container" podUID="055b8496-b4ed-4c0b-9367-c95ba5d4b230" containerName="revoke-keys"
Nov 03 22:00:41.379801 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.379729    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055b8496-b4ed-4c0b-9367-c95ba5d4b230" containerName="revoke-keys"
Nov 03 22:00:41.379850 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.379826    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="055b8496-b4ed-4c0b-9367-c95ba5d4b230" containerName="revoke-keys"
Nov 03 22:00:41.383257 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.383239    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.487386 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.487354    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b598m\" (UniqueName: \"kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.487552 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.487520    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.487627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.487594    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.528308 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.528273    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:00:41.528511 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.528362    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:00:41.530586 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.530565    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"]
Nov 03 22:00:41.588197 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.588157    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.588547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.588226    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b598m\" (UniqueName: \"kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.588547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.588318    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.589293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.589260    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.590917 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.590888    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.644912 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.644826    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b598m\" (UniqueName: \"kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m\") pod \"maas-api-revoke-sub-5eca68c3-td6zz\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:41.697075 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:41.697034    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:42.073041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:42.073011    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"]
Nov 03 22:00:42.075157 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:42.075129    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b794cf_2d34_4536_b3c5_a3017371228c.slice/crio-0ad40345a58999c238d059b98c226f4edb41cf00bccafc61d3d2842fd4c42a37 WatchSource:0}: Error finding container 0ad40345a58999c238d059b98c226f4edb41cf00bccafc61d3d2842fd4c42a37: Status 404 returned error can't find the container with id 0ad40345a58999c238d059b98c226f4edb41cf00bccafc61d3d2842fd4c42a37
Nov 03 22:00:42.885692 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:42.885656    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz" event={"ID":"f2b794cf-2d34-4536-b3c5-a3017371228c","Type":"ContainerStarted","Data":"b747bcbfd2328720ada18948c59956f135d093cb2a879022bd87f534dadf7db7"}
Nov 03 22:00:42.885877 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:42.885710    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz" event={"ID":"f2b794cf-2d34-4536-b3c5-a3017371228c","Type":"ContainerStarted","Data":"0ad40345a58999c238d059b98c226f4edb41cf00bccafc61d3d2842fd4c42a37"}
Nov 03 22:00:42.999957 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:42.999897    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz" podStartSLOduration=1.9998780969999999 podStartE2EDuration="1.999878097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:42.997206629 +0000 UTC m=+2713.981203211" watchObservedRunningTime="2026-11-03 22:00:42.999878097 +0000 UTC m=+2713.983874664"
Nov 03 22:00:43.894380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:43.894321    5244 generic.go:356] "Generic (PLEG): container finished" podID="f2b794cf-2d34-4536-b3c5-a3017371228c" containerID="b747bcbfd2328720ada18948c59956f135d093cb2a879022bd87f534dadf7db7" exitCode=0
Nov 03 22:00:43.894855 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:43.894431    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz" event={"ID":"f2b794cf-2d34-4536-b3c5-a3017371228c","Type":"ContainerDied","Data":"b747bcbfd2328720ada18948c59956f135d093cb2a879022bd87f534dadf7db7"}
Nov 03 22:00:45.050199 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.050167    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:45.118614 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.118573    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca\") pod \"f2b794cf-2d34-4536-b3c5-a3017371228c\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") "
Nov 03 22:00:45.118822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.118629    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m\" (UniqueName: \"kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m\") pod \"f2b794cf-2d34-4536-b3c5-a3017371228c\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") "
Nov 03 22:00:45.118822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.118788    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca\") pod \"f2b794cf-2d34-4536-b3c5-a3017371228c\" (UID: \"f2b794cf-2d34-4536-b3c5-a3017371228c\") "
Nov 03 22:00:45.118953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.118919    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca" pod "f2b794cf-2d34-4536-b3c5-a3017371228c" (UID: "f2b794cf-2d34-4536-b3c5-a3017371228c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:45.119142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.119123    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:45.120822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.120799    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m" pod "f2b794cf-2d34-4536-b3c5-a3017371228c" (UID: "f2b794cf-2d34-4536-b3c5-a3017371228c"). InnerVolumeSpecName "kube-api-access-b598m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:45.120925 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.120876    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca" pod "f2b794cf-2d34-4536-b3c5-a3017371228c" (UID: "f2b794cf-2d34-4536-b3c5-a3017371228c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:00:45.220014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.219977    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b794cf-2d34-4536-b3c5-a3017371228c-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:45.220014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.220012    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b598m\" (UniqueName: \"kubernetes.io/projected/f2b794cf-2d34-4536-b3c5-a3017371228c-kube-api-access-b598m\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:45.538031 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.537952    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"]
Nov 03 22:00:45.538477 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.538462    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f2b794cf-2d34-4536-b3c5-a3017371228c" containerName="revoke-keys"
Nov 03 22:00:45.538534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.538479    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b794cf-2d34-4536-b3c5-a3017371228c" containerName="revoke-keys"
Nov 03 22:00:45.538580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.538561    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b794cf-2d34-4536-b3c5-a3017371228c" containerName="revoke-keys"
Nov 03 22:00:45.541741 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.541724    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623363    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623446    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623484    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623501    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623704 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623642    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fv6g\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623704 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623677    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623704 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623698    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623812 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623727    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.623872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.623845    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.652174 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.652138    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"]
Nov 03 22:00:45.725146 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725111    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725153    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725181    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725217    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725272    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725316    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725370    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725769 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725440    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fv6g\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725769 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725472    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725769 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725634    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725769 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725652    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.725769 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725690    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.726065 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.725924    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.726166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.726145    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.727913 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.727890    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.728157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.728134    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.781858 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.781821    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.782001 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.781963    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fv6g\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.855208 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.855112    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:45.910202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.910081    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz" event={"ID":"f2b794cf-2d34-4536-b3c5-a3017371228c","Type":"ContainerDied","Data":"0ad40345a58999c238d059b98c226f4edb41cf00bccafc61d3d2842fd4c42a37"}
Nov 03 22:00:45.910202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.910152    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad40345a58999c238d059b98c226f4edb41cf00bccafc61d3d2842fd4c42a37"
Nov 03 22:00:45.910393 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:45.910231    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"
Nov 03 22:00:46.286087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:46.286061    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"]
Nov 03 22:00:46.287828 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:46.287804    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2220fc07_2e8a_4427_bade_be4bdd90add3.slice/crio-9c76dee66f09d65663f6ec1355463218b6950718a506038a2afa94889a7d3410 WatchSource:0}: Error finding container 9c76dee66f09d65663f6ec1355463218b6950718a506038a2afa94889a7d3410: Status 404 returned error can't find the container with id 9c76dee66f09d65663f6ec1355463218b6950718a506038a2afa94889a7d3410
Nov 03 22:00:46.290382 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:46.290313    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:00:46.290498 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:46.290430    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:00:46.290498 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:46.290482    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:00:46.918368 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:46.918321    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" event={"ID":"2220fc07-2e8a-4427-bade-be4bdd90add3","Type":"ContainerStarted","Data":"3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1"}
Nov 03 22:00:46.918541 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:46.918389    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" event={"ID":"2220fc07-2e8a-4427-bade-be4bdd90add3","Type":"ContainerStarted","Data":"9c76dee66f09d65663f6ec1355463218b6950718a506038a2afa94889a7d3410"}
Nov 03 22:00:47.856521 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:47.856481    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:47.858537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:47.858504    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 22:00:47.858674 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:47.858573    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 22:00:48.856869 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:48.856828    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 22:00:48.857282 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:48.856893    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 22:00:49.856399 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:49.855781    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 22:00:49.856399 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:49.855887    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 22:00:49.868368 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:49.868323    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:00:49.982692 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:49.982638    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podStartSLOduration=4.982623444 podStartE2EDuration="4.982623444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:00:47.045087658 +0000 UTC m=+2718.029084243" watchObservedRunningTime="2026-11-03 22:00:49.982623444 +0000 UTC m=+2720.966620004"
Nov 03 22:00:50.857355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:50.856811    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 22:00:50.857355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:50.856895    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 22:00:50.875823 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:50.875795    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:00:51.856746 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:51.856705    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 22:00:51.856938 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:51.856770    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 22:00:52.438472 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:00:52.438428    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:00:52.856473 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:52.856385    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 03 22:00:52.856473 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:52.856447    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 03 22:00:53.860924 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:53.860891    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:53.861528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:53.861315    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:53.862030 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:53.862005    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:00:56.026213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.026184    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"]
Nov 03 22:00:56.029864 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.029849    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.132770 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.132736    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:00:56.132974 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.132942    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:00:56.135226 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.135206    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"]
Nov 03 22:00:56.155851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.155823    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.156060 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.156036    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.156134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.156096    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vbb\" (UniqueName: \"kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.256948 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.256911    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.256948 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.256953    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vbb\" (UniqueName: \"kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.257202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.257011    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.257757 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.257729    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.259576 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.259554    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.405868 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.405798    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vbb\" (UniqueName: \"kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb\") pod \"maas-api-revoke-sub-68850315-dlvj2\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:56.642911 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:56.642880    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:57.017374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:57.017347    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"]
Nov 03 22:00:57.019939 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:00:57.019910    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93435a2d_d10a_4910_9e6f_13e79470b181.slice/crio-80ff0252454f936128caa7ee83af9daf0737a915e52831e448a3880705170b15 WatchSource:0}: Error finding container 80ff0252454f936128caa7ee83af9daf0737a915e52831e448a3880705170b15: Status 404 returned error can't find the container with id 80ff0252454f936128caa7ee83af9daf0737a915e52831e448a3880705170b15
Nov 03 22:00:58.017076 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:58.017041    5244 generic.go:356] "Generic (PLEG): container finished" podID="93435a2d-d10a-4910-9e6f-13e79470b181" containerID="8ccf85ba0cf8608c4c6eaf1eb3d4a8bed8ca2f9d312895004252a3035e823480" exitCode=0
Nov 03 22:00:58.017501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:58.017098    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2" event={"ID":"93435a2d-d10a-4910-9e6f-13e79470b181","Type":"ContainerDied","Data":"8ccf85ba0cf8608c4c6eaf1eb3d4a8bed8ca2f9d312895004252a3035e823480"}
Nov 03 22:00:58.017501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:58.017134    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2" event={"ID":"93435a2d-d10a-4910-9e6f-13e79470b181","Type":"ContainerStarted","Data":"80ff0252454f936128caa7ee83af9daf0737a915e52831e448a3880705170b15"}
Nov 03 22:00:59.165289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.165265    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:00:59.316049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.315972    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca\") pod \"93435a2d-d10a-4910-9e6f-13e79470b181\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") "
Nov 03 22:00:59.316341 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.316299    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb\" (UniqueName: \"kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb\") pod \"93435a2d-d10a-4910-9e6f-13e79470b181\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") "
Nov 03 22:00:59.316508 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.316489    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca\") pod \"93435a2d-d10a-4910-9e6f-13e79470b181\" (UID: \"93435a2d-d10a-4910-9e6f-13e79470b181\") "
Nov 03 22:00:59.316581 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.316544    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca" pod "93435a2d-d10a-4910-9e6f-13e79470b181" (UID: "93435a2d-d10a-4910-9e6f-13e79470b181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:00:59.316820 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.316800    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:59.319553 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.319525    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca" pod "93435a2d-d10a-4910-9e6f-13e79470b181" (UID: "93435a2d-d10a-4910-9e6f-13e79470b181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:00:59.321300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.321275    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb" pod "93435a2d-d10a-4910-9e6f-13e79470b181" (UID: "93435a2d-d10a-4910-9e6f-13e79470b181"). InnerVolumeSpecName "kube-api-access-v4vbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:00:59.417781 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.417749    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93435a2d-d10a-4910-9e6f-13e79470b181-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:00:59.417781 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:00:59.417777    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vbb\" (UniqueName: \"kubernetes.io/projected/93435a2d-d10a-4910-9e6f-13e79470b181-kube-api-access-v4vbb\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:00.034916 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:00.032656    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2" event={"ID":"93435a2d-d10a-4910-9e6f-13e79470b181","Type":"ContainerDied","Data":"80ff0252454f936128caa7ee83af9daf0737a915e52831e448a3880705170b15"}
Nov 03 22:01:00.034916 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:00.032719    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ff0252454f936128caa7ee83af9daf0737a915e52831e448a3880705170b15"
Nov 03 22:01:00.034916 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:00.032794    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"
Nov 03 22:01:10.182742 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.182705    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"]
Nov 03 22:01:10.183374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.183348    5244 cpu_manager.go:414] "containerMap: removing container" podUID="93435a2d-d10a-4910-9e6f-13e79470b181" containerName="revoke-keys"
Nov 03 22:01:10.183374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.183373    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93435a2d-d10a-4910-9e6f-13e79470b181" containerName="revoke-keys"
Nov 03 22:01:10.183551 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.183536    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="93435a2d-d10a-4910-9e6f-13e79470b181" containerName="revoke-keys"
Nov 03 22:01:10.186864 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.186845    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.240008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.239978    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:01:10.242903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.242877    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"]
Nov 03 22:01:10.289622 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.289583    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:01:10.336083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.336042    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.336271 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.336197    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.336401 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.336382    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjmg\" (UniqueName: \"kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.437896 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.437808    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjmg\" (UniqueName: \"kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.437896 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.437864    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.438199 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.437919    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.438596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.438573    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.440630 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.440608    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.494216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.494191    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjmg\" (UniqueName: \"kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg\") pod \"maas-api-revoke-sub-92618d08-7hdf4\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.500010 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.499990    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:10.880839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:10.880812    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"]
Nov 03 22:01:10.883193 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:10.883165    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dcbb8d_c1bf_4ce5_bf83_757700a117b9.slice/crio-d8b99555ee09156f238bff00e225355f21506460c29e5b1d379b093f964d27ff WatchSource:0}: Error finding container d8b99555ee09156f238bff00e225355f21506460c29e5b1d379b093f964d27ff: Status 404 returned error can't find the container with id d8b99555ee09156f238bff00e225355f21506460c29e5b1d379b093f964d27ff
Nov 03 22:01:11.098225 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:11.098188    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4" event={"ID":"41dcbb8d-c1bf-4ce5-bf83-757700a117b9","Type":"ContainerStarted","Data":"d8b99555ee09156f238bff00e225355f21506460c29e5b1d379b093f964d27ff"}
Nov 03 22:01:12.105761 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:12.105670    5244 generic.go:356] "Generic (PLEG): container finished" podID="41dcbb8d-c1bf-4ce5-bf83-757700a117b9" containerID="04cc1fd44746fe720b7bf9944702788f8a5cad2e896f84cb47bdbf5d219f3d83" exitCode=0
Nov 03 22:01:12.106211 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:12.105758    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4" event={"ID":"41dcbb8d-c1bf-4ce5-bf83-757700a117b9","Type":"ContainerDied","Data":"04cc1fd44746fe720b7bf9944702788f8a5cad2e896f84cb47bdbf5d219f3d83"}
Nov 03 22:01:13.263426 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.263399    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:13.266469 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.266450    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg\" (UniqueName: \"kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg\") pod \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") "
Nov 03 22:01:13.266573 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.266562    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca\") pod \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") "
Nov 03 22:01:13.266612 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.266604    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca\") pod \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\" (UID: \"41dcbb8d-c1bf-4ce5-bf83-757700a117b9\") "
Nov 03 22:01:13.266922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.266897    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca" pod "41dcbb8d-c1bf-4ce5-bf83-757700a117b9" (UID: "41dcbb8d-c1bf-4ce5-bf83-757700a117b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:01:13.268696 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.268673    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg" pod "41dcbb8d-c1bf-4ce5-bf83-757700a117b9" (UID: "41dcbb8d-c1bf-4ce5-bf83-757700a117b9"). InnerVolumeSpecName "kube-api-access-wzjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:01:13.268773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.268741    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca" pod "41dcbb8d-c1bf-4ce5-bf83-757700a117b9" (UID: "41dcbb8d-c1bf-4ce5-bf83-757700a117b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:01:13.367467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.367383    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:13.367467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.367415    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:13.367467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:13.367428    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzjmg\" (UniqueName: \"kubernetes.io/projected/41dcbb8d-c1bf-4ce5-bf83-757700a117b9-kube-api-access-wzjmg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:14.117852 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:14.117821    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"
Nov 03 22:01:14.117852 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:14.117827    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4" event={"ID":"41dcbb8d-c1bf-4ce5-bf83-757700a117b9","Type":"ContainerDied","Data":"d8b99555ee09156f238bff00e225355f21506460c29e5b1d379b093f964d27ff"}
Nov 03 22:01:14.118087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:14.117872    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b99555ee09156f238bff00e225355f21506460c29e5b1d379b093f964d27ff"
Nov 03 22:01:22.437443 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:01:22.437404    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:01:26.277301 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.277267    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"]
Nov 03 22:01:26.277908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.277890    5244 cpu_manager.go:414] "containerMap: removing container" podUID="41dcbb8d-c1bf-4ce5-bf83-757700a117b9" containerName="revoke-keys"
Nov 03 22:01:26.277964 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.277910    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41dcbb8d-c1bf-4ce5-bf83-757700a117b9" containerName="revoke-keys"
Nov 03 22:01:26.278003 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.277985    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="41dcbb8d-c1bf-4ce5-bf83-757700a117b9" containerName="revoke-keys"
Nov 03 22:01:26.281417 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.281401    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.335361 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.335302    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:01:26.338432 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.338391    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"]
Nov 03 22:01:26.384362 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.384313    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:01:26.415858 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.415820    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.416056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.415886    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbkpg\" (UniqueName: \"kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.416056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.416015    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.517131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.517098    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.517303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.517209    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.517303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.517278    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbkpg\" (UniqueName: \"kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.517999 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.517973    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.519762 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.519745    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.544366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.544281    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:01:26.574201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.574174    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbkpg\" (UniqueName: \"kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg\") pod \"maas-api-revoke-sub-b6072b12-f47s2\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.594487 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.594451    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:26.972624 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:26.972595    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"]
Nov 03 22:01:26.975114 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:26.975080    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a0e87b_56f3_4997_b9c5_372034c836ea.slice/crio-36c3ffbc93b618a94c7e36dbf3f80c39dd31e38b09baecfea110b53c1599c6fe WatchSource:0}: Error finding container 36c3ffbc93b618a94c7e36dbf3f80c39dd31e38b09baecfea110b53c1599c6fe: Status 404 returned error can't find the container with id 36c3ffbc93b618a94c7e36dbf3f80c39dd31e38b09baecfea110b53c1599c6fe
Nov 03 22:01:27.209646 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:27.209604    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2" event={"ID":"b3a0e87b-56f3-4997-b9c5-372034c836ea","Type":"ContainerStarted","Data":"36c3ffbc93b618a94c7e36dbf3f80c39dd31e38b09baecfea110b53c1599c6fe"}
Nov 03 22:01:28.225374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:28.219164    5244 generic.go:356] "Generic (PLEG): container finished" podID="b3a0e87b-56f3-4997-b9c5-372034c836ea" containerID="23d9da974931e012c16d875f8a598d69c3284b7872da7e5d565112c359a958c9" exitCode=0
Nov 03 22:01:28.225374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:28.219237    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2" event={"ID":"b3a0e87b-56f3-4997-b9c5-372034c836ea","Type":"ContainerDied","Data":"23d9da974931e012c16d875f8a598d69c3284b7872da7e5d565112c359a958c9"}
Nov 03 22:01:29.370253 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.370227    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:29.457680 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.457650    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca\") pod \"b3a0e87b-56f3-4997-b9c5-372034c836ea\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") "
Nov 03 22:01:29.457869 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.457759    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca\") pod \"b3a0e87b-56f3-4997-b9c5-372034c836ea\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") "
Nov 03 22:01:29.457869 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.457784    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg\" (UniqueName: \"kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg\") pod \"b3a0e87b-56f3-4997-b9c5-372034c836ea\" (UID: \"b3a0e87b-56f3-4997-b9c5-372034c836ea\") "
Nov 03 22:01:29.458039 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.458015    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca" pod "b3a0e87b-56f3-4997-b9c5-372034c836ea" (UID: "b3a0e87b-56f3-4997-b9c5-372034c836ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:01:29.459994 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.459968    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca" pod "b3a0e87b-56f3-4997-b9c5-372034c836ea" (UID: "b3a0e87b-56f3-4997-b9c5-372034c836ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:01:29.460106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.460043    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg" pod "b3a0e87b-56f3-4997-b9c5-372034c836ea" (UID: "b3a0e87b-56f3-4997-b9c5-372034c836ea"). InnerVolumeSpecName "kube-api-access-kbkpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:01:29.559617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.559513    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:29.559617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.559546    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3a0e87b-56f3-4997-b9c5-372034c836ea-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:29.559617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:29.559555    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbkpg\" (UniqueName: \"kubernetes.io/projected/b3a0e87b-56f3-4997-b9c5-372034c836ea-kube-api-access-kbkpg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:30.231795 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:30.231763    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"
Nov 03 22:01:30.231980 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:30.231793    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2" event={"ID":"b3a0e87b-56f3-4997-b9c5-372034c836ea","Type":"ContainerDied","Data":"36c3ffbc93b618a94c7e36dbf3f80c39dd31e38b09baecfea110b53c1599c6fe"}
Nov 03 22:01:30.231980 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:30.231866    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36c3ffbc93b618a94c7e36dbf3f80c39dd31e38b09baecfea110b53c1599c6fe"
Nov 03 22:01:32.217951 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.217917    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"]
Nov 03 22:01:32.219177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.219147    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b3a0e87b-56f3-4997-b9c5-372034c836ea" containerName="revoke-keys"
Nov 03 22:01:32.219177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.219178    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a0e87b-56f3-4997-b9c5-372034c836ea" containerName="revoke-keys"
Nov 03 22:01:32.219434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.219294    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a0e87b-56f3-4997-b9c5-372034c836ea" containerName="revoke-keys"
Nov 03 22:01:32.222955 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.222867    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.277709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.277671    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"]
Nov 03 22:01:32.288722 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.288687    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.288900 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.288754    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.288900 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.288881    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hjk\" (UniqueName: \"kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.323907 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.323878    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:01:32.323907 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.323906    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:01:32.389922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.389885    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.390122 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.389949    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.390122 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.390061    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hjk\" (UniqueName: \"kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.390623 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.390600    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.392572 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.392549    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.492796 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.492720    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hjk\" (UniqueName: \"kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk\") pod \"maas-api-revoke-sub-3c2d9624-7skzd\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.536601 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.536564    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:32.912243 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:32.912213    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"]
Nov 03 22:01:32.913257 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:32.913232    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5e1231_6905_4b9f_a7f1_f5876dbc7926.slice/crio-701af5ecb3ff8d846b8991e8e15e97304dfc02176ff3f2480dfb6c799593fe37 WatchSource:0}: Error finding container 701af5ecb3ff8d846b8991e8e15e97304dfc02176ff3f2480dfb6c799593fe37: Status 404 returned error can't find the container with id 701af5ecb3ff8d846b8991e8e15e97304dfc02176ff3f2480dfb6c799593fe37
Nov 03 22:01:33.250534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:33.250501    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd" event={"ID":"1b5e1231-6905-4b9f-a7f1-f5876dbc7926","Type":"ContainerStarted","Data":"701af5ecb3ff8d846b8991e8e15e97304dfc02176ff3f2480dfb6c799593fe37"}
Nov 03 22:01:34.266122 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:34.266086    5244 generic.go:356] "Generic (PLEG): container finished" podID="1b5e1231-6905-4b9f-a7f1-f5876dbc7926" containerID="c3dd3b23e4f6a0c8f12a30498eddecd982b57963c35e29b4111d367dc2321e1a" exitCode=0
Nov 03 22:01:34.266638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:34.266176    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd" event={"ID":"1b5e1231-6905-4b9f-a7f1-f5876dbc7926","Type":"ContainerDied","Data":"c3dd3b23e4f6a0c8f12a30498eddecd982b57963c35e29b4111d367dc2321e1a"}
Nov 03 22:01:35.442169 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.442142    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:35.525790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.525695    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca\") pod \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") "
Nov 03 22:01:35.525979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.525793    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk\" (UniqueName: \"kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk\") pod \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") "
Nov 03 22:01:35.525979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.525877    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca\") pod \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\" (UID: \"1b5e1231-6905-4b9f-a7f1-f5876dbc7926\") "
Nov 03 22:01:35.526097 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.526062    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca" pod "1b5e1231-6905-4b9f-a7f1-f5876dbc7926" (UID: "1b5e1231-6905-4b9f-a7f1-f5876dbc7926"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:01:35.526318 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.526298    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:35.528017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.527996    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk" pod "1b5e1231-6905-4b9f-a7f1-f5876dbc7926" (UID: "1b5e1231-6905-4b9f-a7f1-f5876dbc7926"). InnerVolumeSpecName "kube-api-access-49hjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:01:35.528158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.528140    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca" pod "1b5e1231-6905-4b9f-a7f1-f5876dbc7926" (UID: "1b5e1231-6905-4b9f-a7f1-f5876dbc7926"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:01:35.627606 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.627563    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hjk\" (UniqueName: \"kubernetes.io/projected/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-kube-api-access-49hjk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:35.627606 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:35.627599    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b5e1231-6905-4b9f-a7f1-f5876dbc7926-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:36.278707 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:36.278677    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"
Nov 03 22:01:36.278886 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:36.278712    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd" event={"ID":"1b5e1231-6905-4b9f-a7f1-f5876dbc7926","Type":"ContainerDied","Data":"701af5ecb3ff8d846b8991e8e15e97304dfc02176ff3f2480dfb6c799593fe37"}
Nov 03 22:01:36.278886 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:36.278760    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="701af5ecb3ff8d846b8991e8e15e97304dfc02176ff3f2480dfb6c799593fe37"
Nov 03 22:01:39.501532 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.501498    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"]
Nov 03 22:01:39.502003 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.501988    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1b5e1231-6905-4b9f-a7f1-f5876dbc7926" containerName="revoke-keys"
Nov 03 22:01:39.502078 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.502005    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5e1231-6905-4b9f-a7f1-f5876dbc7926" containerName="revoke-keys"
Nov 03 22:01:39.502152 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.502139    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5e1231-6905-4b9f-a7f1-f5876dbc7926" containerName="revoke-keys"
Nov 03 22:01:39.505651 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.505635    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.572997 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.572963    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.572997 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573001    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573022    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573112    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573235    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573269    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573347    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573419    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.573500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.573484    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w666n\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.598555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.598519    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"]
Nov 03 22:01:39.674854 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.674818    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.674865    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.674893    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.674929    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w666n\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.674987    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675000    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675026    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675055    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675131    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675690 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675589    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675690 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675608    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675690 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675673    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675895 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675692    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.675895 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.675705    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.677744 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.677723    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.677850 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.677838    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.731953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.731916    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.732134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.732077    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w666n\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:39.819953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:39.819839    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:40.251093 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:40.251070    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"]
Nov 03 22:01:40.256634 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:40.256602    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93233c9c_b516_4ca1_97e1_42dcabe8e4c0.slice/crio-9e8e705f793b29dafb4c5167adc77b41009170ed7ea9b7d5b904a79bfe375f95 WatchSource:0}: Error finding container 9e8e705f793b29dafb4c5167adc77b41009170ed7ea9b7d5b904a79bfe375f95: Status 404 returned error can't find the container with id 9e8e705f793b29dafb4c5167adc77b41009170ed7ea9b7d5b904a79bfe375f95
Nov 03 22:01:40.259397 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:40.259363    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:01:40.259468 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:40.259433    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:01:40.259512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:40.259467    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:01:40.323905 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:40.323863    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" event={"ID":"93233c9c-b516-4ca1-97e1-42dcabe8e4c0","Type":"ContainerStarted","Data":"9e8e705f793b29dafb4c5167adc77b41009170ed7ea9b7d5b904a79bfe375f95"}
Nov 03 22:01:41.331210 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.331171    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" event={"ID":"93233c9c-b516-4ca1-97e1-42dcabe8e4c0","Type":"ContainerStarted","Data":"ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf"}
Nov 03 22:01:41.572094 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.572035    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podStartSLOduration=2.572016745 podStartE2EDuration="2.572016745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:41.521097521 +0000 UTC m=+2772.505094106" watchObservedRunningTime="2026-11-03 22:01:41.572016745 +0000 UTC m=+2772.556013304"
Nov 03 22:01:41.600802 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.600729    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"]
Nov 03 22:01:41.604455 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.604439    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.659347 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.659294    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:01:41.659624 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.659402    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:01:41.663361 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.663308    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"]
Nov 03 22:01:41.699239 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.699202    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.699406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.699260    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.699466 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.699442    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4phm\" (UniqueName: \"kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.800570 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.800535    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.800753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.800593    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.800753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.800693    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4phm\" (UniqueName: \"kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.801356 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.801307    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.803059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.803033    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.821061 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.821037    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:41.822588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.822567    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 22:01:41.822645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.822615    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 22:01:41.859541 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.859455    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4phm\" (UniqueName: \"kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm\") pod \"maas-api-revoke-sub-6b90f921-wxnsz\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:41.918409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:41.918375    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:42.328775 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:42.328744    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"]
Nov 03 22:01:42.330306 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:42.330276    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623e22a1_7707_493b_90fc_662e7ae15f9d.slice/crio-1fd0bce46d620f7e646bbe0dfe886682ac4f65d9dab1369f95a137e38f92a91a WatchSource:0}: Error finding container 1fd0bce46d620f7e646bbe0dfe886682ac4f65d9dab1369f95a137e38f92a91a: Status 404 returned error can't find the container with id 1fd0bce46d620f7e646bbe0dfe886682ac4f65d9dab1369f95a137e38f92a91a
Nov 03 22:01:42.378218 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:42.378187    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz" event={"ID":"623e22a1-7707-493b-90fc-662e7ae15f9d","Type":"ContainerStarted","Data":"1fd0bce46d620f7e646bbe0dfe886682ac4f65d9dab1369f95a137e38f92a91a"}
Nov 03 22:01:42.821494 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:42.821454    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 22:01:42.821675 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:42.821531    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 22:01:43.386100 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:43.386062    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz" event={"ID":"623e22a1-7707-493b-90fc-662e7ae15f9d","Type":"ContainerStarted","Data":"d9e18f984d8b161ce7a49c2e48d23ccf7b26f0008c73f3d0aab0e5d7229a61ec"}
Nov 03 22:01:43.503537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:43.503468    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz" podStartSLOduration=2.503452703 podStartE2EDuration="2.503452703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:43.500674115 +0000 UTC m=+2774.484670692" watchObservedRunningTime="2026-11-03 22:01:43.503452703 +0000 UTC m=+2774.487449261"
Nov 03 22:01:43.820532 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:43.820495    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 22:01:43.820731 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:43.820555    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 22:01:44.394063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:44.394025    5244 generic.go:356] "Generic (PLEG): container finished" podID="623e22a1-7707-493b-90fc-662e7ae15f9d" containerID="d9e18f984d8b161ce7a49c2e48d23ccf7b26f0008c73f3d0aab0e5d7229a61ec" exitCode=0
Nov 03 22:01:44.394523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:44.394104    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz" event={"ID":"623e22a1-7707-493b-90fc-662e7ae15f9d","Type":"ContainerDied","Data":"d9e18f984d8b161ce7a49c2e48d23ccf7b26f0008c73f3d0aab0e5d7229a61ec"}
Nov 03 22:01:44.820195 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:44.820160    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 22:01:44.820425 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:44.820227    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 22:01:45.587943 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.587919    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:45.681201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.681101    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm\" (UniqueName: \"kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm\") pod \"623e22a1-7707-493b-90fc-662e7ae15f9d\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") "
Nov 03 22:01:45.681201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.681147    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca\") pod \"623e22a1-7707-493b-90fc-662e7ae15f9d\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") "
Nov 03 22:01:45.681201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.681177    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca\") pod \"623e22a1-7707-493b-90fc-662e7ae15f9d\" (UID: \"623e22a1-7707-493b-90fc-662e7ae15f9d\") "
Nov 03 22:01:45.681557 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.681496    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca" pod "623e22a1-7707-493b-90fc-662e7ae15f9d" (UID: "623e22a1-7707-493b-90fc-662e7ae15f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:01:45.681666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.681649    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:45.683402 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.683380    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm" pod "623e22a1-7707-493b-90fc-662e7ae15f9d" (UID: "623e22a1-7707-493b-90fc-662e7ae15f9d"). InnerVolumeSpecName "kube-api-access-z4phm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:01:45.683480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.683449    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca" pod "623e22a1-7707-493b-90fc-662e7ae15f9d" (UID: "623e22a1-7707-493b-90fc-662e7ae15f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:01:45.783309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.783265    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4phm\" (UniqueName: \"kubernetes.io/projected/623e22a1-7707-493b-90fc-662e7ae15f9d-kube-api-access-z4phm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:45.783309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.783301    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623e22a1-7707-493b-90fc-662e7ae15f9d-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:45.821158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.821121    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 22:01:45.821370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:45.821174    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 22:01:46.406680 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:46.406648    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"
Nov 03 22:01:46.406876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:46.406691    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz" event={"ID":"623e22a1-7707-493b-90fc-662e7ae15f9d","Type":"ContainerDied","Data":"1fd0bce46d620f7e646bbe0dfe886682ac4f65d9dab1369f95a137e38f92a91a"}
Nov 03 22:01:46.406876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:46.406751    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd0bce46d620f7e646bbe0dfe886682ac4f65d9dab1369f95a137e38f92a91a"
Nov 03 22:01:46.820656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:46.820612    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Nov 03 22:01:46.821161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:46.820680    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Nov 03 22:01:47.833231 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.833199    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:47.833724 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.833587    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:47.837666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.837644    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:01:47.878220 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.878184    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"]
Nov 03 22:01:47.878887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.878867    5244 cpu_manager.go:414] "containerMap: removing container" podUID="623e22a1-7707-493b-90fc-662e7ae15f9d" containerName="revoke-keys"
Nov 03 22:01:47.878887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.878888    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623e22a1-7707-493b-90fc-662e7ae15f9d" containerName="revoke-keys"
Nov 03 22:01:47.879072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.879010    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="623e22a1-7707-493b-90fc-662e7ae15f9d" containerName="revoke-keys"
Nov 03 22:01:47.886758 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.886725    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:47.938381 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.938322    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"]
Nov 03 22:01:47.938880 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.938854    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:01:47.938880 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:47.938877    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:01:48.008441 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.008401    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjg9p\" (UniqueName: \"kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.008617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.008452    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.008681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.008668    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.114636 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.114514    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.114853 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.114677    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjg9p\" (UniqueName: \"kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.114853 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.114711    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.116053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.116027    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.119276 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.119243    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.222485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.222449    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjg9p\" (UniqueName: \"kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p\") pod \"maas-api-revoke-sub-ba5fb38d-cv62c\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.507732 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.507512    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:48.921090 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:48.921057    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"]
Nov 03 22:01:48.921554 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:48.921529    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c1cc68_ebea_4c36_b1f3_afc0b02c12e1.slice/crio-2927bf1c5bda07a735cbad087090ba231e16d17157c9d9b7b3d905e6ccdab1f1 WatchSource:0}: Error finding container 2927bf1c5bda07a735cbad087090ba231e16d17157c9d9b7b3d905e6ccdab1f1: Status 404 returned error can't find the container with id 2927bf1c5bda07a735cbad087090ba231e16d17157c9d9b7b3d905e6ccdab1f1
Nov 03 22:01:49.438815 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:49.438776    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c" event={"ID":"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1","Type":"ContainerStarted","Data":"cdfde7cba099c4e54415a68b7f9a107e5ba3ef5a2a4bf6529b1ba910453486c2"}
Nov 03 22:01:49.439028 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:49.438847    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c" event={"ID":"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1","Type":"ContainerStarted","Data":"2927bf1c5bda07a735cbad087090ba231e16d17157c9d9b7b3d905e6ccdab1f1"}
Nov 03 22:01:49.552504 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:49.552395    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c" podStartSLOduration=2.552379331 podStartE2EDuration="2.552379331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:01:49.550260309 +0000 UTC m=+2780.534256879" watchObservedRunningTime="2026-11-03 22:01:49.552379331 +0000 UTC m=+2780.536375894"
Nov 03 22:01:50.447622 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:50.447580    5244 generic.go:356] "Generic (PLEG): container finished" podID="e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" containerID="cdfde7cba099c4e54415a68b7f9a107e5ba3ef5a2a4bf6529b1ba910453486c2" exitCode=0
Nov 03 22:01:50.448108 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:50.447670    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c" event={"ID":"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1","Type":"ContainerDied","Data":"cdfde7cba099c4e54415a68b7f9a107e5ba3ef5a2a4bf6529b1ba910453486c2"}
Nov 03 22:01:51.588993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.588964    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:51.687853 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.687821    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca\") pod \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") "
Nov 03 22:01:51.688017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.687906    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca\") pod \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") "
Nov 03 22:01:51.688017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.687982    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p\" (UniqueName: \"kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p\") pod \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\" (UID: \"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1\") "
Nov 03 22:01:51.688291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.688264    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca" pod "e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" (UID: "e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:01:51.690228 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.690204    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p" pod "e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" (UID: "e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1"). InnerVolumeSpecName "kube-api-access-zjg9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:01:51.690228 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.690210    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca" pod "e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" (UID: "e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:01:51.790979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.790882    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjg9p\" (UniqueName: \"kubernetes.io/projected/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-kube-api-access-zjg9p\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:51.790979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.790923    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:51.790979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:51.790939    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:52.436470 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:01:52.436427    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:01:52.461290 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:52.461257    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"
Nov 03 22:01:52.461523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:52.461297    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c" event={"ID":"e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1","Type":"ContainerDied","Data":"2927bf1c5bda07a735cbad087090ba231e16d17157c9d9b7b3d905e6ccdab1f1"}
Nov 03 22:01:52.461523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:52.461371    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2927bf1c5bda07a735cbad087090ba231e16d17157c9d9b7b3d905e6ccdab1f1"
Nov 03 22:01:56.766455 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.766410    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"]
Nov 03 22:01:56.766967 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.766952    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" containerName="revoke-keys"
Nov 03 22:01:56.767030 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.766971    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" containerName="revoke-keys"
Nov 03 22:01:56.767079 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.767068    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" containerName="revoke-keys"
Nov 03 22:01:56.770047 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.770032    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.827981 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.827938    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"]
Nov 03 22:01:56.846380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.846316    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zk4\" (UniqueName: \"kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.846617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.846590    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.846663 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.846646    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.873545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.873512    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:01:56.873729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.873557    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:01:56.947655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.947624    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.947655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.947665    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.947875 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.947743    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zk4\" (UniqueName: \"kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.948494 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.948460    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:56.950176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:56.950153    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:57.004468 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:57.004435    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zk4\" (UniqueName: \"kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4\") pod \"maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:57.082717 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:57.082631    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:57.462034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:57.461996    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"]
Nov 03 22:01:57.463676 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:01:57.463647    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee4b0c4_da02_4430_9807_41bbfbed55d9.slice/crio-4b67421645b3997abbe3d524d9a1dfc68cf5c325e2094be2fa2eebc6274ef694 WatchSource:0}: Error finding container 4b67421645b3997abbe3d524d9a1dfc68cf5c325e2094be2fa2eebc6274ef694: Status 404 returned error can't find the container with id 4b67421645b3997abbe3d524d9a1dfc68cf5c325e2094be2fa2eebc6274ef694
Nov 03 22:01:57.503761 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:57.503728    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5" event={"ID":"9ee4b0c4-da02-4430-9807-41bbfbed55d9","Type":"ContainerStarted","Data":"4b67421645b3997abbe3d524d9a1dfc68cf5c325e2094be2fa2eebc6274ef694"}
Nov 03 22:01:58.511648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:58.511613    5244 generic.go:356] "Generic (PLEG): container finished" podID="9ee4b0c4-da02-4430-9807-41bbfbed55d9" containerID="7891e2b2a38fd70cfe81f2d8d7b30ddb0b8243de80974bd8e6b6cb91e593814d" exitCode=0
Nov 03 22:01:58.512085 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:58.511657    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5" event={"ID":"9ee4b0c4-da02-4430-9807-41bbfbed55d9","Type":"ContainerDied","Data":"7891e2b2a38fd70cfe81f2d8d7b30ddb0b8243de80974bd8e6b6cb91e593814d"}
Nov 03 22:01:59.674627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.674598    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:01:59.779432 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.779311    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca\") pod \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") "
Nov 03 22:01:59.779627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.779433    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4\" (UniqueName: \"kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4\") pod \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") "
Nov 03 22:01:59.779627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.779462    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca\") pod \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\" (UID: \"9ee4b0c4-da02-4430-9807-41bbfbed55d9\") "
Nov 03 22:01:59.779753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.779685    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca" pod "9ee4b0c4-da02-4430-9807-41bbfbed55d9" (UID: "9ee4b0c4-da02-4430-9807-41bbfbed55d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:01:59.779857 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.779842    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:59.781764 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.781739    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4" pod "9ee4b0c4-da02-4430-9807-41bbfbed55d9" (UID: "9ee4b0c4-da02-4430-9807-41bbfbed55d9"). InnerVolumeSpecName "kube-api-access-65zk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:01:59.781971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.781957    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca" pod "9ee4b0c4-da02-4430-9807-41bbfbed55d9" (UID: "9ee4b0c4-da02-4430-9807-41bbfbed55d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:01:59.881172 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.881132    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zk4\" (UniqueName: \"kubernetes.io/projected/9ee4b0c4-da02-4430-9807-41bbfbed55d9-kube-api-access-65zk4\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:01:59.881172 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:01:59.881168    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee4b0c4-da02-4430-9807-41bbfbed55d9-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:00.526819 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:00.526786    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5" event={"ID":"9ee4b0c4-da02-4430-9807-41bbfbed55d9","Type":"ContainerDied","Data":"4b67421645b3997abbe3d524d9a1dfc68cf5c325e2094be2fa2eebc6274ef694"}
Nov 03 22:02:00.526993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:00.526837    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b67421645b3997abbe3d524d9a1dfc68cf5c325e2094be2fa2eebc6274ef694"
Nov 03 22:02:00.526993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:00.526800    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"
Nov 03 22:02:04.304028 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.303984    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"]
Nov 03 22:02:04.304570 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.304488    5244 cpu_manager.go:414] "containerMap: removing container" podUID="9ee4b0c4-da02-4430-9807-41bbfbed55d9" containerName="revoke-keys"
Nov 03 22:02:04.304570 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.304502    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee4b0c4-da02-4430-9807-41bbfbed55d9" containerName="revoke-keys"
Nov 03 22:02:04.304687 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.304600    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee4b0c4-da02-4430-9807-41bbfbed55d9" containerName="revoke-keys"
Nov 03 22:02:04.308043 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.308025    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.366388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.366352    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"]
Nov 03 22:02:04.411962 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.411925    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:04.411962 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.411960    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:04.431949 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.431919    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.432143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.432042    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzqf\" (UniqueName: \"kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.432143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.432093    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.533178 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.533141    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.533406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.533232    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzqf\" (UniqueName: \"kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.533406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.533258    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.533872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.533845    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.535761 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.535736    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.590092 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.590004    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzqf\" (UniqueName: \"kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf\") pod \"maas-api-revoke-sub-f68f3ba8-stkd8\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.620720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.620683    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:04.998898 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:04.998860    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"]
Nov 03 22:02:05.000088 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:05.000049    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1044aafb_0b47_4c51_9dba_467b8d960124.slice/crio-37752f1e3419f0a4aefd804fa7681af6fce8fe6d7a3201817f7d3137ea2c308b WatchSource:0}: Error finding container 37752f1e3419f0a4aefd804fa7681af6fce8fe6d7a3201817f7d3137ea2c308b: Status 404 returned error can't find the container with id 37752f1e3419f0a4aefd804fa7681af6fce8fe6d7a3201817f7d3137ea2c308b
Nov 03 22:02:05.559728 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:05.559691    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8" event={"ID":"1044aafb-0b47-4c51-9dba-467b8d960124","Type":"ContainerStarted","Data":"bb750400dbe349ad54a952ad2fc38eceb4af7f800e228675d487fb608fab588a"}
Nov 03 22:02:05.560158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:05.559761    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8" event={"ID":"1044aafb-0b47-4c51-9dba-467b8d960124","Type":"ContainerStarted","Data":"37752f1e3419f0a4aefd804fa7681af6fce8fe6d7a3201817f7d3137ea2c308b"}
Nov 03 22:02:05.673346 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:05.673277    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8" podStartSLOduration=1.673262331 podStartE2EDuration="1.673262331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:05.6708863 +0000 UTC m=+2796.654882859" watchObservedRunningTime="2026-11-03 22:02:05.673262331 +0000 UTC m=+2796.657258889"
Nov 03 22:02:06.567140 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:06.567056    5244 generic.go:356] "Generic (PLEG): container finished" podID="1044aafb-0b47-4c51-9dba-467b8d960124" containerID="bb750400dbe349ad54a952ad2fc38eceb4af7f800e228675d487fb608fab588a" exitCode=0
Nov 03 22:02:06.567561 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:06.567136    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8" event={"ID":"1044aafb-0b47-4c51-9dba-467b8d960124","Type":"ContainerDied","Data":"bb750400dbe349ad54a952ad2fc38eceb4af7f800e228675d487fb608fab588a"}
Nov 03 22:02:07.711068 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.711039    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:07.770681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.770642    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf\" (UniqueName: \"kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf\") pod \"1044aafb-0b47-4c51-9dba-467b8d960124\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") "
Nov 03 22:02:07.770860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.770767    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca\") pod \"1044aafb-0b47-4c51-9dba-467b8d960124\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") "
Nov 03 22:02:07.770908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.770876    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca\") pod \"1044aafb-0b47-4c51-9dba-467b8d960124\" (UID: \"1044aafb-0b47-4c51-9dba-467b8d960124\") "
Nov 03 22:02:07.771114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.771089    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca" pod "1044aafb-0b47-4c51-9dba-467b8d960124" (UID: "1044aafb-0b47-4c51-9dba-467b8d960124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:07.771270 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.771249    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:07.773044 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.773018    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf" pod "1044aafb-0b47-4c51-9dba-467b8d960124" (UID: "1044aafb-0b47-4c51-9dba-467b8d960124"). InnerVolumeSpecName "kube-api-access-npzqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:07.773141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.773058    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca" pod "1044aafb-0b47-4c51-9dba-467b8d960124" (UID: "1044aafb-0b47-4c51-9dba-467b8d960124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:07.872979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.872897    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npzqf\" (UniqueName: \"kubernetes.io/projected/1044aafb-0b47-4c51-9dba-467b8d960124-kube-api-access-npzqf\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:07.872979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:07.872930    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1044aafb-0b47-4c51-9dba-467b8d960124-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:08.579411 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:08.579380    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"
Nov 03 22:02:08.579411 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:08.579397    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8" event={"ID":"1044aafb-0b47-4c51-9dba-467b8d960124","Type":"ContainerDied","Data":"37752f1e3419f0a4aefd804fa7681af6fce8fe6d7a3201817f7d3137ea2c308b"}
Nov 03 22:02:08.579629 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:08.579481    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37752f1e3419f0a4aefd804fa7681af6fce8fe6d7a3201817f7d3137ea2c308b"
Nov 03 22:02:10.320610 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.320571    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"]
Nov 03 22:02:10.321298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.321276    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1044aafb-0b47-4c51-9dba-467b8d960124" containerName="revoke-keys"
Nov 03 22:02:10.321366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.321301    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1044aafb-0b47-4c51-9dba-467b8d960124" containerName="revoke-keys"
Nov 03 22:02:10.321535 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.321522    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1044aafb-0b47-4c51-9dba-467b8d960124" containerName="revoke-keys"
Nov 03 22:02:10.325505 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.325487    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.380597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.380565    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"]
Nov 03 22:02:10.413194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.413145    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szs6b\" (UniqueName: \"kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.413461 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.413382    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.413461 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.413426    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.425482 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.425456    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:10.425623 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.425506    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:10.514908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.514865    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.515115 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.514933    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.515115 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.515042    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szs6b\" (UniqueName: \"kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.515655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.515626    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.517540 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.517517    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.572025 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.571947    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szs6b\" (UniqueName: \"kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b\") pod \"maas-api-revoke-sub-5eea77b4-rs9pk\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:10.641357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:10.641297    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:11.017726 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:11.017690    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"]
Nov 03 22:02:11.019318 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:11.019290    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2b3dc4_2815_4a89_a9b8_606e37173768.slice/crio-21285e11cad47aefeee9076653f3137ec034dbfecf71835480aa89c5a55aa286 WatchSource:0}: Error finding container 21285e11cad47aefeee9076653f3137ec034dbfecf71835480aa89c5a55aa286: Status 404 returned error can't find the container with id 21285e11cad47aefeee9076653f3137ec034dbfecf71835480aa89c5a55aa286
Nov 03 22:02:11.633205 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:11.633104    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk" event={"ID":"5d2b3dc4-2815-4a89-a9b8-606e37173768","Type":"ContainerStarted","Data":"a2537c9e0f2bf660ca2d378c0edd6f86fcf9fd816ceb0f9d924391963cc368fc"}
Nov 03 22:02:11.633205 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:11.633176    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk" event={"ID":"5d2b3dc4-2815-4a89-a9b8-606e37173768","Type":"ContainerStarted","Data":"21285e11cad47aefeee9076653f3137ec034dbfecf71835480aa89c5a55aa286"}
Nov 03 22:02:11.747161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:11.747103    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk" podStartSLOduration=1.747088773 podStartE2EDuration="1.747088773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:11.745911792 +0000 UTC m=+2802.729908373" watchObservedRunningTime="2026-11-03 22:02:11.747088773 +0000 UTC m=+2802.731085331"
Nov 03 22:02:12.641009 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:12.640969    5244 generic.go:356] "Generic (PLEG): container finished" podID="5d2b3dc4-2815-4a89-a9b8-606e37173768" containerID="a2537c9e0f2bf660ca2d378c0edd6f86fcf9fd816ceb0f9d924391963cc368fc" exitCode=0
Nov 03 22:02:12.641547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:12.641053    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk" event={"ID":"5d2b3dc4-2815-4a89-a9b8-606e37173768","Type":"ContainerDied","Data":"a2537c9e0f2bf660ca2d378c0edd6f86fcf9fd816ceb0f9d924391963cc368fc"}
Nov 03 22:02:13.790216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.790192    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:13.953422 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.953374    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca\") pod \"5d2b3dc4-2815-4a89-a9b8-606e37173768\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") "
Nov 03 22:02:13.953580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.953529    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b\" (UniqueName: \"kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b\") pod \"5d2b3dc4-2815-4a89-a9b8-606e37173768\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") "
Nov 03 22:02:13.953580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.953557    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca\") pod \"5d2b3dc4-2815-4a89-a9b8-606e37173768\" (UID: \"5d2b3dc4-2815-4a89-a9b8-606e37173768\") "
Nov 03 22:02:13.953959 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.953933    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca" pod "5d2b3dc4-2815-4a89-a9b8-606e37173768" (UID: "5d2b3dc4-2815-4a89-a9b8-606e37173768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:13.955615 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.955592    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca" pod "5d2b3dc4-2815-4a89-a9b8-606e37173768" (UID: "5d2b3dc4-2815-4a89-a9b8-606e37173768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:13.955775 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:13.955752    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b" pod "5d2b3dc4-2815-4a89-a9b8-606e37173768" (UID: "5d2b3dc4-2815-4a89-a9b8-606e37173768"). InnerVolumeSpecName "kube-api-access-szs6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:14.054773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:14.054738    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:14.054773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:14.054768    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b3dc4-2815-4a89-a9b8-606e37173768-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:14.054773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:14.054779    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szs6b\" (UniqueName: \"kubernetes.io/projected/5d2b3dc4-2815-4a89-a9b8-606e37173768-kube-api-access-szs6b\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:14.654545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:14.654510    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk" event={"ID":"5d2b3dc4-2815-4a89-a9b8-606e37173768","Type":"ContainerDied","Data":"21285e11cad47aefeee9076653f3137ec034dbfecf71835480aa89c5a55aa286"}
Nov 03 22:02:14.654545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:14.654547    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"
Nov 03 22:02:14.654765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:14.654564    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21285e11cad47aefeee9076653f3137ec034dbfecf71835480aa89c5a55aa286"
Nov 03 22:02:16.046106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:16.046071    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"]
Nov 03 22:02:16.095352 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:16.095291    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848b7cd6-tbv5q"]
Nov 03 22:02:17.135788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:17.135750    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8115f0-811d-4085-b736-651b8c0f9309" path="/var/lib/kubelet/pods/7d8115f0-811d-4085-b736-651b8c0f9309/volumes"
Nov 03 22:02:18.439910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.439879    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"]
Nov 03 22:02:18.440412 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.440398    5244 cpu_manager.go:414] "containerMap: removing container" podUID="5d2b3dc4-2815-4a89-a9b8-606e37173768" containerName="revoke-keys"
Nov 03 22:02:18.440474 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.440413    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2b3dc4-2815-4a89-a9b8-606e37173768" containerName="revoke-keys"
Nov 03 22:02:18.440511 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.440486    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2b3dc4-2815-4a89-a9b8-606e37173768" containerName="revoke-keys"
Nov 03 22:02:18.443755 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.443733    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.501478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.501264    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"]
Nov 03 22:02:18.503411 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.503378    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.503579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.503456    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lg6\" (UniqueName: \"kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.503579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.503501    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.543673 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.543640    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:18.545252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.545234    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:18.605105 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.605071    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.605314 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.605126    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lg6\" (UniqueName: \"kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.605314 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.605158    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.605843 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.605817    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.607749 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.607725    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.661593 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.661553    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lg6\" (UniqueName: \"kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6\") pod \"maas-api-revoke-sub-2d58dfae-x9ccs\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:18.757239 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:18.757145    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:19.134049 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:19.134019    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda318f787_035e_4ce0_b268_40c0b357c776.slice/crio-576bc91310840c9a9f199aa2070acf0a41bf9059d6552f9a674a44f2e60fc07e WatchSource:0}: Error finding container 576bc91310840c9a9f199aa2070acf0a41bf9059d6552f9a674a44f2e60fc07e: Status 404 returned error can't find the container with id 576bc91310840c9a9f199aa2070acf0a41bf9059d6552f9a674a44f2e60fc07e
Nov 03 22:02:19.137309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:19.137287    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"]
Nov 03 22:02:19.707414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:19.706424    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs" event={"ID":"a318f787-035e-4ce0-b268-40c0b357c776","Type":"ContainerStarted","Data":"5fe79f9cb6a3ec9301bece13ed0f0f467da0af84df1c98afa4c2db143e8e7dea"}
Nov 03 22:02:19.707414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:19.706496    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs" event={"ID":"a318f787-035e-4ce0-b268-40c0b357c776","Type":"ContainerStarted","Data":"576bc91310840c9a9f199aa2070acf0a41bf9059d6552f9a674a44f2e60fc07e"}
Nov 03 22:02:19.823565 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:19.823504    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs" podStartSLOduration=1.823489431 podStartE2EDuration="1.823489431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:19.820659816 +0000 UTC m=+2810.804656394" watchObservedRunningTime="2026-11-03 22:02:19.823489431 +0000 UTC m=+2810.807486023"
Nov 03 22:02:20.039043 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:20.039008    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:02:20.714342 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:20.714292    5244 generic.go:356] "Generic (PLEG): container finished" podID="a318f787-035e-4ce0-b268-40c0b357c776" containerID="5fe79f9cb6a3ec9301bece13ed0f0f467da0af84df1c98afa4c2db143e8e7dea" exitCode=0
Nov 03 22:02:20.714750 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:20.714386    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs" event={"ID":"a318f787-035e-4ce0-b268-40c0b357c776","Type":"ContainerDied","Data":"5fe79f9cb6a3ec9301bece13ed0f0f467da0af84df1c98afa4c2db143e8e7dea"}
Nov 03 22:02:22.440434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.422092    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:22.469633 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:02:22.469578    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:02:22.511287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.511247    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca\") pod \"a318f787-035e-4ce0-b268-40c0b357c776\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") "
Nov 03 22:02:22.511470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.511397    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca\") pod \"a318f787-035e-4ce0-b268-40c0b357c776\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") "
Nov 03 22:02:22.511470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.511430    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6\" (UniqueName: \"kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6\") pod \"a318f787-035e-4ce0-b268-40c0b357c776\" (UID: \"a318f787-035e-4ce0-b268-40c0b357c776\") "
Nov 03 22:02:22.511819 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.511788    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca" pod "a318f787-035e-4ce0-b268-40c0b357c776" (UID: "a318f787-035e-4ce0-b268-40c0b357c776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:22.513685 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.513656    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6" pod "a318f787-035e-4ce0-b268-40c0b357c776" (UID: "a318f787-035e-4ce0-b268-40c0b357c776"). InnerVolumeSpecName "kube-api-access-c2lg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:22.513783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.513702    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca" pod "a318f787-035e-4ce0-b268-40c0b357c776" (UID: "a318f787-035e-4ce0-b268-40c0b357c776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:22.612807 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.612768    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:22.612807 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.612800    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2lg6\" (UniqueName: \"kubernetes.io/projected/a318f787-035e-4ce0-b268-40c0b357c776-kube-api-access-c2lg6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:22.613024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.612817    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a318f787-035e-4ce0-b268-40c0b357c776-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:22.728872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.728792    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"
Nov 03 22:02:22.728872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.728832    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs" event={"ID":"a318f787-035e-4ce0-b268-40c0b357c776","Type":"ContainerDied","Data":"576bc91310840c9a9f199aa2070acf0a41bf9059d6552f9a674a44f2e60fc07e"}
Nov 03 22:02:22.729083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:22.728916    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="576bc91310840c9a9f199aa2070acf0a41bf9059d6552f9a674a44f2e60fc07e"
Nov 03 22:02:25.549126 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.549093    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"]
Nov 03 22:02:25.549646 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.549631    5244 cpu_manager.go:414] "containerMap: removing container" podUID="a318f787-035e-4ce0-b268-40c0b357c776" containerName="revoke-keys"
Nov 03 22:02:25.549703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.549647    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a318f787-035e-4ce0-b268-40c0b357c776" containerName="revoke-keys"
Nov 03 22:02:25.549762 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.549752    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="a318f787-035e-4ce0-b268-40c0b357c776" containerName="revoke-keys"
Nov 03 22:02:25.553019 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.553002    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.643221 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.643177    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.643433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.643353    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlfq\" (UniqueName: \"kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.643433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.643396    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.655182 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.655157    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:25.655182 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.655179    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:25.658059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.658039    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"]
Nov 03 22:02:25.744760 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.744720    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlfq\" (UniqueName: \"kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.744760 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.744765    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.745023 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.744854    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.745655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.745633    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.747323 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.747302    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.801364 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.801271    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlfq\" (UniqueName: \"kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:25.866653 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:25.866621    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:26.243755 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:26.243723    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"]
Nov 03 22:02:26.245475 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:26.245447    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda54f1cb_2cb4_4c8e_a84f_b939af7b9359.slice/crio-cfdb0dc37dc1b8a1ca885f8492096191da5274948e415cd27cd6a8114d7fa550 WatchSource:0}: Error finding container cfdb0dc37dc1b8a1ca885f8492096191da5274948e415cd27cd6a8114d7fa550: Status 404 returned error can't find the container with id cfdb0dc37dc1b8a1ca885f8492096191da5274948e415cd27cd6a8114d7fa550
Nov 03 22:02:26.755408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:26.755364    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z" event={"ID":"da54f1cb-2cb4-4c8e-a84f-b939af7b9359","Type":"ContainerStarted","Data":"b9a037c56f1712f9635648250382db94322be208484dc7b661caf7bd6eea93e7"}
Nov 03 22:02:26.755827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:26.755462    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z" event={"ID":"da54f1cb-2cb4-4c8e-a84f-b939af7b9359","Type":"ContainerStarted","Data":"cfdb0dc37dc1b8a1ca885f8492096191da5274948e415cd27cd6a8114d7fa550"}
Nov 03 22:02:26.900403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:26.900263    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z" podStartSLOduration=1.9002462709999999 podStartE2EDuration="1.900246271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:26.899280509 +0000 UTC m=+2817.883277069" watchObservedRunningTime="2026-11-03 22:02:26.900246271 +0000 UTC m=+2817.884242921"
Nov 03 22:02:27.357285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:27.357247    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"]
Nov 03 22:02:27.357516 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:27.357491    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" containerID="cri-o://047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7" gracePeriod=30
Nov 03 22:02:27.762540 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:27.762500    5244 generic.go:356] "Generic (PLEG): container finished" podID="da54f1cb-2cb4-4c8e-a84f-b939af7b9359" containerID="b9a037c56f1712f9635648250382db94322be208484dc7b661caf7bd6eea93e7" exitCode=0
Nov 03 22:02:27.763044 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:27.762585    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z" event={"ID":"da54f1cb-2cb4-4c8e-a84f-b939af7b9359","Type":"ContainerDied","Data":"b9a037c56f1712f9635648250382db94322be208484dc7b661caf7bd6eea93e7"}
Nov 03 22:02:28.231630 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.231592    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"]
Nov 03 22:02:28.235497 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.235481    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.276842 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.276809    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.276842 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.276846    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkq8\" (UniqueName: \"kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.277100 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.277073    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.291875 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.291844    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"]
Nov 03 22:02:28.378207 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.378170    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.378406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.378247    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.378406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.378269    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkq8\" (UniqueName: \"kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.378914 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.378892    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.380618 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.380598    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.434524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.434489    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkq8\" (UniqueName: \"kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8\") pod \"maas-api-revoke-sub-a4cc6926-58ksl\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.548974 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.548893    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:28.915632 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.915598    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:28.923499 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.923473    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"]
Nov 03 22:02:28.924504 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:28.924467    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ecd850_8a21_4ba0_bbdf_d0bd18809ea4.slice/crio-71888511d6f0d4179d7cc2d60e5500a1c74ce338e38110f45765c8618d21ee44 WatchSource:0}: Error finding container 71888511d6f0d4179d7cc2d60e5500a1c74ce338e38110f45765c8618d21ee44: Status 404 returned error can't find the container with id 71888511d6f0d4179d7cc2d60e5500a1c74ce338e38110f45765c8618d21ee44
Nov 03 22:02:28.985015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.984985    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca\") pod \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") "
Nov 03 22:02:28.985202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.985023    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca\") pod \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") "
Nov 03 22:02:28.985202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.985138    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq\" (UniqueName: \"kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq\") pod \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\" (UID: \"da54f1cb-2cb4-4c8e-a84f-b939af7b9359\") "
Nov 03 22:02:28.985527 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.985501    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca" pod "da54f1cb-2cb4-4c8e-a84f-b939af7b9359" (UID: "da54f1cb-2cb4-4c8e-a84f-b939af7b9359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:28.987348 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.987307    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca" pod "da54f1cb-2cb4-4c8e-a84f-b939af7b9359" (UID: "da54f1cb-2cb4-4c8e-a84f-b939af7b9359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:28.987480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:28.987369    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq" pod "da54f1cb-2cb4-4c8e-a84f-b939af7b9359" (UID: "da54f1cb-2cb4-4c8e-a84f-b939af7b9359"). InnerVolumeSpecName "kube-api-access-fzlfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:29.086638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.086542    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzlfq\" (UniqueName: \"kubernetes.io/projected/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-kube-api-access-fzlfq\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:29.086638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.086570    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:29.086638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.086585    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da54f1cb-2cb4-4c8e-a84f-b939af7b9359-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:29.218801 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.218760    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"]
Nov 03 22:02:29.219530 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.219503    5244 cpu_manager.go:414] "containerMap: removing container" podUID="da54f1cb-2cb4-4c8e-a84f-b939af7b9359" containerName="revoke-keys"
Nov 03 22:02:29.219530 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.219526    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da54f1cb-2cb4-4c8e-a84f-b939af7b9359" containerName="revoke-keys"
Nov 03 22:02:29.219738 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.219669    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="da54f1cb-2cb4-4c8e-a84f-b939af7b9359" containerName="revoke-keys"
Nov 03 22:02:29.224880 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.224858    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.279346 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.279299    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"]
Nov 03 22:02:29.288889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.288860    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.289077 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.288930    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.289077 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.289049    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9s2\" (UniqueName: \"kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.301491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.301458    5244 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5nhq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:02:29.301650 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.301515    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:02:29.390583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.390551    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.390743 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.390654    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.390743 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.390728    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9s2\" (UniqueName: \"kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.391476 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.391454    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.393174 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.393154    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.455079 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.455047    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9s2\" (UniqueName: \"kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2\") pod \"maas-api-revoke-sub-f1257e9a-hlkr7\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.540864 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.540823    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:29.779788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.779762    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"
Nov 03 22:02:29.779788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.779769    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z" event={"ID":"da54f1cb-2cb4-4c8e-a84f-b939af7b9359","Type":"ContainerDied","Data":"cfdb0dc37dc1b8a1ca885f8492096191da5274948e415cd27cd6a8114d7fa550"}
Nov 03 22:02:29.780063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.779853    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfdb0dc37dc1b8a1ca885f8492096191da5274948e415cd27cd6a8114d7fa550"
Nov 03 22:02:29.781530 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.781497    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl" event={"ID":"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4","Type":"ContainerStarted","Data":"ebd1a969ef5804e51a51c71fc0de54b3fc9ce27efcd151b5742f3e6ed3e2d0a7"}
Nov 03 22:02:29.781672 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.781551    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl" event={"ID":"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4","Type":"ContainerStarted","Data":"71888511d6f0d4179d7cc2d60e5500a1c74ce338e38110f45765c8618d21ee44"}
Nov 03 22:02:29.895900 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.895846    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl" podStartSLOduration=1.895831351 podStartE2EDuration="1.895831351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:29.892412969 +0000 UTC m=+2820.876409558" watchObservedRunningTime="2026-11-03 22:02:29.895831351 +0000 UTC m=+2820.879827909"
Nov 03 22:02:29.932894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.932857    5244 scope.go:122] "RemoveContainer" containerID="ada473276d10afd74cb9d2fa49ef308e16c7d46a0de28931ba811115c04b332f"
Nov 03 22:02:29.967084 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:29.967057    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"]
Nov 03 22:02:29.968102 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:29.968075    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6318b399_f69d_4817_9980_b9ec438771de.slice/crio-6b82eebd7d62bbb6c3c9e3ac2fc138d74a489b45fd2596788db2724e8dac4d72 WatchSource:0}: Error finding container 6b82eebd7d62bbb6c3c9e3ac2fc138d74a489b45fd2596788db2724e8dac4d72: Status 404 returned error can't find the container with id 6b82eebd7d62bbb6c3c9e3ac2fc138d74a489b45fd2596788db2724e8dac4d72
Nov 03 22:02:30.789020 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:30.788977    5244 generic.go:356] "Generic (PLEG): container finished" podID="36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" containerID="ebd1a969ef5804e51a51c71fc0de54b3fc9ce27efcd151b5742f3e6ed3e2d0a7" exitCode=0
Nov 03 22:02:30.789231 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:30.789051    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl" event={"ID":"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4","Type":"ContainerDied","Data":"ebd1a969ef5804e51a51c71fc0de54b3fc9ce27efcd151b5742f3e6ed3e2d0a7"}
Nov 03 22:02:30.790731 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:30.790705    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7" event={"ID":"6318b399-f69d-4817-9980-b9ec438771de","Type":"ContainerStarted","Data":"1bc645ef2020fb755b4e3893a837857e393a463e83d9ae4d87ce5896656c5ed5"}
Nov 03 22:02:30.790873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:30.790748    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7" event={"ID":"6318b399-f69d-4817-9980-b9ec438771de","Type":"ContainerStarted","Data":"6b82eebd7d62bbb6c3c9e3ac2fc138d74a489b45fd2596788db2724e8dac4d72"}
Nov 03 22:02:31.013297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:31.013228    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7" podStartSLOduration=2.013212122 podStartE2EDuration="2.013212122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:31.012660604 +0000 UTC m=+2821.996657185" watchObservedRunningTime="2026-11-03 22:02:31.013212122 +0000 UTC m=+2821.997208677"
Nov 03 22:02:31.798410 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:31.798371    5244 generic.go:356] "Generic (PLEG): container finished" podID="6318b399-f69d-4817-9980-b9ec438771de" containerID="1bc645ef2020fb755b4e3893a837857e393a463e83d9ae4d87ce5896656c5ed5" exitCode=0
Nov 03 22:02:31.798599 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:31.798490    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7" event={"ID":"6318b399-f69d-4817-9980-b9ec438771de","Type":"ContainerDied","Data":"1bc645ef2020fb755b4e3893a837857e393a463e83d9ae4d87ce5896656c5ed5"}
Nov 03 22:02:31.955186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:31.955159    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:32.031027 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.030986    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8\" (UniqueName: \"kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8\") pod \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") "
Nov 03 22:02:32.031559 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.031086    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca\") pod \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") "
Nov 03 22:02:32.031559 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.031107    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca\") pod \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\" (UID: \"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4\") "
Nov 03 22:02:32.031559 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.031504    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca" pod "36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" (UID: "36ecd850-8a21-4ba0-bbdf-d0bd18809ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:32.033451 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.033423    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca" pod "36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" (UID: "36ecd850-8a21-4ba0-bbdf-d0bd18809ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:32.033747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.033714    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8" pod "36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" (UID: "36ecd850-8a21-4ba0-bbdf-d0bd18809ea4"). InnerVolumeSpecName "kube-api-access-bjkq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:32.132300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.132205    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjkq8\" (UniqueName: \"kubernetes.io/projected/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-kube-api-access-bjkq8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.132300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.132233    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.132300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.132244    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.602534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.602507    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 22:02:32.739876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.739790    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.739876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.739852    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.739888    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.739942    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.739991    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740032    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740072    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740092    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:02:32.740550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740165    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740207    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data\") pod \"e9fab12b-73aa-4812-b607-d27434308bc2\" (UID: \"e9fab12b-73aa-4812-b607-d27434308bc2\") "
Nov 03 22:02:32.740550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740388    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:02:32.740550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740432    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:02:32.740550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740447    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:32.740824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740621    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:02:32.740824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740717    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fab12b-73aa-4812-b607-d27434308bc2-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.740824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740755    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.740824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740772    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.740824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740786    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.740824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.740801    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.742305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.742275    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "kube-api-access-w4clw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:32.742625 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.742603    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:02:32.742790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.742769    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:02:32.742790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.742779    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token" pod "e9fab12b-73aa-4812-b607-d27434308bc2" (UID: "e9fab12b-73aa-4812-b607-d27434308bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:32.807779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.807745    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"
Nov 03 22:02:32.807968 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.807782    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl" event={"ID":"36ecd850-8a21-4ba0-bbdf-d0bd18809ea4","Type":"ContainerDied","Data":"71888511d6f0d4179d7cc2d60e5500a1c74ce338e38110f45765c8618d21ee44"}
Nov 03 22:02:32.807968 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.807850    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71888511d6f0d4179d7cc2d60e5500a1c74ce338e38110f45765c8618d21ee44"
Nov 03 22:02:32.809528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.809498    5244 generic.go:356] "Generic (PLEG): container finished" podID="e9fab12b-73aa-4812-b607-d27434308bc2" containerID="047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7" exitCode=0
Nov 03 22:02:32.809681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.809543    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"
Nov 03 22:02:32.809681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.809583    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" event={"ID":"e9fab12b-73aa-4812-b607-d27434308bc2","Type":"ContainerDied","Data":"047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7"}
Nov 03 22:02:32.809681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.809636    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4" event={"ID":"e9fab12b-73aa-4812-b607-d27434308bc2","Type":"ContainerDied","Data":"16d596b7fd10a9f930882fe9fdb96cb1eb8c015f5e8e773093fba549f54570f2"}
Nov 03 22:02:32.809681 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.809667    5244 scope.go:122] "RemoveContainer" containerID="047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7"
Nov 03 22:02:32.842285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.842223    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fab12b-73aa-4812-b607-d27434308bc2-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.842285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.842258    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fab12b-73aa-4812-b607-d27434308bc2-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.842285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.842272    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4clw\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-kube-api-access-w4clw\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.842285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.842285    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fab12b-73aa-4812-b607-d27434308bc2-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:32.842866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.842837    5244 scope.go:122] "RemoveContainer" containerID="047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7"
Nov 03 22:02:32.843541 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:02:32.843517    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7\": container with ID starting with 047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7 not found: ID does not exist" containerID="047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7"
Nov 03 22:02:32.843630 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.843550    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7"} err="failed to get container status \"047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7\": rpc error: code = NotFound desc = could not find container \"047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7\": container with ID starting with 047a9537ab9b7fec68c0f56bbf17026fa8524c47a6234423016fefe5c0979ca7 not found: ID does not exist"
Nov 03 22:02:32.971912 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.971884    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:32.973298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.973273    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"]
Nov 03 22:02:32.975864 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:32.975844    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5nhq4"]
Nov 03 22:02:33.044645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.044565    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca\") pod \"6318b399-f69d-4817-9980-b9ec438771de\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") "
Nov 03 22:02:33.044645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.044637    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2\" (UniqueName: \"kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2\") pod \"6318b399-f69d-4817-9980-b9ec438771de\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") "
Nov 03 22:02:33.045152 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.044681    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca\") pod \"6318b399-f69d-4817-9980-b9ec438771de\" (UID: \"6318b399-f69d-4817-9980-b9ec438771de\") "
Nov 03 22:02:33.045152 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.045097    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca" pod "6318b399-f69d-4817-9980-b9ec438771de" (UID: "6318b399-f69d-4817-9980-b9ec438771de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:33.046839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.046818    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2" pod "6318b399-f69d-4817-9980-b9ec438771de" (UID: "6318b399-f69d-4817-9980-b9ec438771de"). InnerVolumeSpecName "kube-api-access-rx9s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:33.046956 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.046876    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca" pod "6318b399-f69d-4817-9980-b9ec438771de" (UID: "6318b399-f69d-4817-9980-b9ec438771de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:33.136586 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.136549    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" path="/var/lib/kubelet/pods/e9fab12b-73aa-4812-b607-d27434308bc2/volumes"
Nov 03 22:02:33.146148 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.146098    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6318b399-f69d-4817-9980-b9ec438771de-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:33.146148 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.146139    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9s2\" (UniqueName: \"kubernetes.io/projected/6318b399-f69d-4817-9980-b9ec438771de-kube-api-access-rx9s2\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:33.146392 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.146155    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6318b399-f69d-4817-9980-b9ec438771de-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:33.817624 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.817584    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"
Nov 03 22:02:33.817624 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.817616    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7" event={"ID":"6318b399-f69d-4817-9980-b9ec438771de","Type":"ContainerDied","Data":"6b82eebd7d62bbb6c3c9e3ac2fc138d74a489b45fd2596788db2724e8dac4d72"}
Nov 03 22:02:33.817840 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:33.817664    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b82eebd7d62bbb6c3c9e3ac2fc138d74a489b45fd2596788db2724e8dac4d72"
Nov 03 22:02:36.535724 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:36.535691    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"]
Nov 03 22:02:36.646118 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:36.646078    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"]
Nov 03 22:02:36.952083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:36.952034    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"]
Nov 03 22:02:36.955513 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:36.955489    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-93c833-hd2g5"]
Nov 03 22:02:37.143548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:37.143513    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee4b0c4-da02-4430-9807-41bbfbed55d9" path="/var/lib/kubelet/pods/9ee4b0c4-da02-4430-9807-41bbfbed55d9/volumes"
Nov 03 22:02:39.969507 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969468    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"]
Nov 03 22:02:39.969928 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969908    5244 cpu_manager.go:414] "containerMap: removing container" podUID="36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" containerName="revoke-keys"
Nov 03 22:02:39.969928 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969919    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" containerName="revoke-keys"
Nov 03 22:02:39.970002 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969950    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy"
Nov 03 22:02:39.970002 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969955    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy"
Nov 03 22:02:39.970002 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969970    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6318b399-f69d-4817-9980-b9ec438771de" containerName="revoke-keys"
Nov 03 22:02:39.970002 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.969976    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6318b399-f69d-4817-9980-b9ec438771de" containerName="revoke-keys"
Nov 03 22:02:39.970124 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.970054    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fab12b-73aa-4812-b607-d27434308bc2" containerName="istio-proxy"
Nov 03 22:02:39.970124 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.970064    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6318b399-f69d-4817-9980-b9ec438771de" containerName="revoke-keys"
Nov 03 22:02:39.970124 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.970071    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" containerName="revoke-keys"
Nov 03 22:02:39.974618 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:39.974589    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157425 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157378    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157425 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157439    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157542    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157601    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157767 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157700    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994nl\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157834 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157816    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157893 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157854    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157942 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157910    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.157942 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.157932    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.177678 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.177650    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"]
Nov 03 22:02:40.258664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258585    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.258664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258627    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.258664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258659    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.258973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258687    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.258973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258745    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.258973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258789    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259377 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.258829    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259454    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259158    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259214    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259456    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259004    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259552    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994nl\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.259747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.259665    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.261356 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.261311    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.261639 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.261621    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.316904 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.316869    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.317092 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.317008    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-994nl\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl\") pod \"e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") " pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:40.587790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:40.587689    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:41.085761 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.085722    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"]
Nov 03 22:02:41.113304 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:41.113273    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68c8bfa_aca0_4316_843e_780c5e898259.slice/crio-947c2e41625cb796e90bdc4500873f7078705d1a80af7ed708d6f5a4b188036b WatchSource:0}: Error finding container 947c2e41625cb796e90bdc4500873f7078705d1a80af7ed708d6f5a4b188036b: Status 404 returned error can't find the container with id 947c2e41625cb796e90bdc4500873f7078705d1a80af7ed708d6f5a4b188036b
Nov 03 22:02:41.115944 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.115902    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:41.116034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.115981    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:41.116034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.116012    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:41.230911 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.230869    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"]
Nov 03 22:02:41.256357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.239164    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.275844    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.275906    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.275930    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lz28\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.275963    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.276001    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.276051    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.276118    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.276177    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.276457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.276255    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.323355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.319820    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"]
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379299    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379382    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379433    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379497    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379532    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379555    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379573    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lz28\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379597    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.379626    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.380034    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.380231    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.380479    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.381107    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.383395 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.381305    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.408798 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.408704    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-29dbd9-openshift-default-dockercfg-fl229\""
Nov 03 22:02:41.421132 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.421102    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.421474 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.421453    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.466434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.457001    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lz28\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.466434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.457237    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token\") pod \"e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") " pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.536782 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.536703    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerName="payload-processing" containerID="cri-o://b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54" gracePeriod=25
Nov 03 22:02:41.586996 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.586951    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:41.647425 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.647294    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" podUID="fe290523-3644-4f1c-8870-63743b82d309" containerName="payload-pre-processing" containerID="cri-o://55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1" gracePeriod=25
Nov 03 22:02:41.876153 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.876111    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" event={"ID":"b68c8bfa-aca0-4316-843e-780c5e898259","Type":"ContainerStarted","Data":"47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a"}
Nov 03 22:02:41.876470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.876449    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" event={"ID":"b68c8bfa-aca0-4316-843e-780c5e898259","Type":"ContainerStarted","Data":"947c2e41625cb796e90bdc4500873f7078705d1a80af7ed708d6f5a4b188036b"}
Nov 03 22:02:41.996318 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:41.996260    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podStartSLOduration=2.996244768 podStartE2EDuration="2.996244768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:41.992709013 +0000 UTC m=+2832.976705572" watchObservedRunningTime="2026-11-03 22:02:41.996244768 +0000 UTC m=+2832.980241327"
Nov 03 22:02:42.081706 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:42.081674    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca5f268_88b9_4c99_a0e8_fcf382698cab.slice/crio-0988acfe5db107476f320346a529c038e9672c92d528a21443f83422f4c3b098 WatchSource:0}: Error finding container 0988acfe5db107476f320346a529c038e9672c92d528a21443f83422f4c3b098: Status 404 returned error can't find the container with id 0988acfe5db107476f320346a529c038e9672c92d528a21443f83422f4c3b098
Nov 03 22:02:42.082268 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.082240    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"]
Nov 03 22:02:42.084768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.084727    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:42.084856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.084836    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:42.084906 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.084884    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:42.587666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.587624    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" podUID="fe290523-3644-4f1c-8870-63743b82d309" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:02:42.588114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.587641    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:02:42.588114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.587855    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:42.589535 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.589508    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 22:02:42.589657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.589553    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 22:02:42.883882 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.883799    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" event={"ID":"0ca5f268-88b9-4c99-a0e8-fcf382698cab","Type":"ContainerStarted","Data":"41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"}
Nov 03 22:02:42.883882 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:42.883853    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" event={"ID":"0ca5f268-88b9-4c99-a0e8-fcf382698cab","Type":"ContainerStarted","Data":"0988acfe5db107476f320346a529c038e9672c92d528a21443f83422f4c3b098"}
Nov 03 22:02:43.010894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.010836    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podStartSLOduration=2.010822812 podStartE2EDuration="2.010822812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:43.008633093 +0000 UTC m=+2833.992629675" watchObservedRunningTime="2026-11-03 22:02:43.010822812 +0000 UTC m=+2833.994819370"
Nov 03 22:02:43.587195 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.587160    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:43.588485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.588464    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 22:02:43.588802 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.588513    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 22:02:43.593502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.593478    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:43.890858 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.890760    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:43.891607 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:43.891586    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:02:44.361555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.361509    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"]
Nov 03 22:02:44.366229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.366204    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.419698 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.419639    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:44.419913 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.419828    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:44.424051 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.424020    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"]
Nov 03 22:02:44.426931 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.426906    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.427063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.426963    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjxd\" (UniqueName: \"kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.427130 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.427055    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.528421 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.528392    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.528581 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.528520    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.528581 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.528574    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjxd\" (UniqueName: \"kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.529177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.529157    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.531208 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.531179    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.585046 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.585009    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjxd\" (UniqueName: \"kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd\") pod \"maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:44.588621 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.588588    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 22:02:44.589051 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.588642    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 22:02:44.683956 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:44.683920    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:45.084335 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:45.084299    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"]
Nov 03 22:02:45.086001 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:45.085970    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebedbc8_30c8_4059_a5d9_f4f94e7f830e.slice/crio-4d84ebf1035562ce21f7fdb74251b351192cb708947fdfd967dd9e3ca27d1809 WatchSource:0}: Error finding container 4d84ebf1035562ce21f7fdb74251b351192cb708947fdfd967dd9e3ca27d1809: Status 404 returned error can't find the container with id 4d84ebf1035562ce21f7fdb74251b351192cb708947fdfd967dd9e3ca27d1809
Nov 03 22:02:45.589269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:45.589228    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 22:02:45.589786 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:45.589288    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 22:02:45.905926 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:45.905826    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz" event={"ID":"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e","Type":"ContainerStarted","Data":"c0db690058673fab82c88bc4b5d7c43cf9b127da28b95744769505e501b5106e"}
Nov 03 22:02:45.905926 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:45.905905    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz" event={"ID":"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e","Type":"ContainerStarted","Data":"4d84ebf1035562ce21f7fdb74251b351192cb708947fdfd967dd9e3ca27d1809"}
Nov 03 22:02:46.022282 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.022221    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz" podStartSLOduration=2.022205384 podStartE2EDuration="2.022205384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:46.019092702 +0000 UTC m=+2837.003089277" watchObservedRunningTime="2026-11-03 22:02:46.022205384 +0000 UTC m=+2837.006201944"
Nov 03 22:02:46.446246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.446151    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"]
Nov 03 22:02:46.464617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.464584    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.468049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.468010    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"]
Nov 03 22:02:46.473720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.473697    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.562350 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.555868    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"]
Nov 03 22:02:46.584382 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584339    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584571 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584420    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584571 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584472    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584571 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584550    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584710 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584579    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.584710 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584601    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584710 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584700    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lg5l\" (UniqueName: \"kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.584827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584745    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.584866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584845    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584909 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584879    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.584909 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584905    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.585004 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.584924    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhl6b\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.588562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.588536    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 22:02:46.588651 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.588590    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 22:02:46.598468 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.598434    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"]
Nov 03 22:02:46.685836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.685808    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.686015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.685845    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.685876    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lg5l\" (UniqueName: \"kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.686015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.685907    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.686015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.685951    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.685979    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686000    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686020    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhl6b\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686065    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686082    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686106    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686134    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686910    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.686940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.686930    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.687128 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.687044    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.687128 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.687112    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.687290 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.687270    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.687377 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.687285    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.689171 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.689149    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.689600 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.689575    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.690066 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.690050    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.759316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.759227    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lg5l\" (UniqueName: \"kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l\") pod \"maas-api-revoke-sub-f028c9e1-ncd4d\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.759316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.759226    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhl6b\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.759832 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.759807    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token\") pod \"e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") " pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.780866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.780837    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:46.791510 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.791484    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:46.923099 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.923066    5244 generic.go:356] "Generic (PLEG): container finished" podID="5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" containerID="c0db690058673fab82c88bc4b5d7c43cf9b127da28b95744769505e501b5106e" exitCode=0
Nov 03 22:02:46.923265 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:46.923221    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz" event={"ID":"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e","Type":"ContainerDied","Data":"c0db690058673fab82c88bc4b5d7c43cf9b127da28b95744769505e501b5106e"}
Nov 03 22:02:47.185490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.185461    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"]
Nov 03 22:02:47.195702 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:02:47.195671    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdaa3d96_17f8_4296_a8e8_cfe87797ad52.slice/crio-1aa7ec7823647fbc4d1ecc34c0bc29eee8bb1420afe512b697a2b8f05aef860e WatchSource:0}: Error finding container 1aa7ec7823647fbc4d1ecc34c0bc29eee8bb1420afe512b697a2b8f05aef860e: Status 404 returned error can't find the container with id 1aa7ec7823647fbc4d1ecc34c0bc29eee8bb1420afe512b697a2b8f05aef860e
Nov 03 22:02:47.304579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.304168    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"]
Nov 03 22:02:47.306177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.306145    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:47.306238 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.306224    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:47.306299 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.306287    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:02:47.589054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.589001    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 22:02:47.589240 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.589083    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 22:02:47.933005 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.932970    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" event={"ID":"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0","Type":"ContainerStarted","Data":"084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc"}
Nov 03 22:02:47.933480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.933036    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" event={"ID":"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0","Type":"ContainerStarted","Data":"77c47bad53251f06fbb19e9cfbe1c867b83e2fe4d108cb42af17d4162ac4ac95"}
Nov 03 22:02:47.935281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.935241    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d" event={"ID":"cdaa3d96-17f8-4296-a8e8-cfe87797ad52","Type":"ContainerStarted","Data":"4ad3152d895924c798dba2487306fa3327aac860522356362a5e82d15b8d5dac"}
Nov 03 22:02:47.935450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:47.935301    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d" event={"ID":"cdaa3d96-17f8-4296-a8e8-cfe87797ad52","Type":"ContainerStarted","Data":"1aa7ec7823647fbc4d1ecc34c0bc29eee8bb1420afe512b697a2b8f05aef860e"}
Nov 03 22:02:48.060527 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.057685    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podStartSLOduration=2.057670109 podStartE2EDuration="2.057670109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:48.052971437 +0000 UTC m=+2839.036967998" watchObservedRunningTime="2026-11-03 22:02:48.057670109 +0000 UTC m=+2839.041666667"
Nov 03 22:02:48.103259 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.103219    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:48.169485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.169430    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d" podStartSLOduration=2.16941721 podStartE2EDuration="2.16941721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:48.166591364 +0000 UTC m=+2839.150587915" watchObservedRunningTime="2026-11-03 22:02:48.16941721 +0000 UTC m=+2839.153413769"
Nov 03 22:02:48.204246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.204138    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca\") pod \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") "
Nov 03 22:02:48.204246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.204187    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca\") pod \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") "
Nov 03 22:02:48.204650 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.204288    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd\" (UniqueName: \"kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd\") pod \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\" (UID: \"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e\") "
Nov 03 22:02:48.204722 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.204643    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca" pod "5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" (UID: "5ebedbc8-30c8-4059-a5d9-f4f94e7f830e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:48.205176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.205151    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:48.206572 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.206541    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca" pod "5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" (UID: "5ebedbc8-30c8-4059-a5d9-f4f94e7f830e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:48.206660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.206581    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd" pod "5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" (UID: "5ebedbc8-30c8-4059-a5d9-f4f94e7f830e"). InnerVolumeSpecName "kube-api-access-5gjxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:48.307114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.307075    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjxd\" (UniqueName: \"kubernetes.io/projected/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-kube-api-access-5gjxd\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:48.307280 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.307120    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:48.792302 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.792257    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:48.794207 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.794160    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 22:02:48.794396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.794270    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 22:02:48.918127 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.918100    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:48.918505 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.918483    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:48.919532 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.919513    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:02:48.963501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.962952    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz" event={"ID":"5ebedbc8-30c8-4059-a5d9-f4f94e7f830e","Type":"ContainerDied","Data":"4d84ebf1035562ce21f7fdb74251b351192cb708947fdfd967dd9e3ca27d1809"}
Nov 03 22:02:48.963501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.963022    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d84ebf1035562ce21f7fdb74251b351192cb708947fdfd967dd9e3ca27d1809"
Nov 03 22:02:48.963501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.963104    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"
Nov 03 22:02:48.966793 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.966756    5244 generic.go:356] "Generic (PLEG): container finished" podID="cdaa3d96-17f8-4296-a8e8-cfe87797ad52" containerID="4ad3152d895924c798dba2487306fa3327aac860522356362a5e82d15b8d5dac" exitCode=0
Nov 03 22:02:48.966946 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:48.966900    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d" event={"ID":"cdaa3d96-17f8-4296-a8e8-cfe87797ad52","Type":"ContainerDied","Data":"4ad3152d895924c798dba2487306fa3327aac860522356362a5e82d15b8d5dac"}
Nov 03 22:02:49.792869 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:49.792832    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 22:02:49.793029 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:49.792889    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 22:02:50.222360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.222302    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:50.363316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.363216    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca\") pod \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") "
Nov 03 22:02:50.363507 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.363464    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca\") pod \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") "
Nov 03 22:02:50.363601 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.363507    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l\" (UniqueName: \"kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l\") pod \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\" (UID: \"cdaa3d96-17f8-4296-a8e8-cfe87797ad52\") "
Nov 03 22:02:50.364479 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.364439    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca" pod "cdaa3d96-17f8-4296-a8e8-cfe87797ad52" (UID: "cdaa3d96-17f8-4296-a8e8-cfe87797ad52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:50.366622 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.366596    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca" pod "cdaa3d96-17f8-4296-a8e8-cfe87797ad52" (UID: "cdaa3d96-17f8-4296-a8e8-cfe87797ad52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:50.366931 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.366907    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l" pod "cdaa3d96-17f8-4296-a8e8-cfe87797ad52" (UID: "cdaa3d96-17f8-4296-a8e8-cfe87797ad52"). InnerVolumeSpecName "kube-api-access-8lg5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:50.465191 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.465149    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:50.465191 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.465183    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lg5l\" (UniqueName: \"kubernetes.io/projected/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-kube-api-access-8lg5l\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:50.465438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.465208    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdaa3d96-17f8-4296-a8e8-cfe87797ad52-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:50.792141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.792105    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 22:02:50.792313 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.792160    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 22:02:50.983190 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.983153    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"
Nov 03 22:02:50.983406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.983156    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d" event={"ID":"cdaa3d96-17f8-4296-a8e8-cfe87797ad52","Type":"ContainerDied","Data":"1aa7ec7823647fbc4d1ecc34c0bc29eee8bb1420afe512b697a2b8f05aef860e"}
Nov 03 22:02:50.983406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:50.983288    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa7ec7823647fbc4d1ecc34c0bc29eee8bb1420afe512b697a2b8f05aef860e"
Nov 03 22:02:51.792668 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:51.792625    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 22:02:51.793060 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:51.792695    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 22:02:52.320379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320319    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"]
Nov 03 22:02:52.320858 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320844    5244 cpu_manager.go:414] "containerMap: removing container" podUID="cdaa3d96-17f8-4296-a8e8-cfe87797ad52" containerName="revoke-keys"
Nov 03 22:02:52.320915 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320860    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdaa3d96-17f8-4296-a8e8-cfe87797ad52" containerName="revoke-keys"
Nov 03 22:02:52.320915 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320888    5244 cpu_manager.go:414] "containerMap: removing container" podUID="5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" containerName="revoke-keys"
Nov 03 22:02:52.320915 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320893    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" containerName="revoke-keys"
Nov 03 22:02:52.321018 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320981    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" containerName="revoke-keys"
Nov 03 22:02:52.321018 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.320994    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdaa3d96-17f8-4296-a8e8-cfe87797ad52" containerName="revoke-keys"
Nov 03 22:02:52.324624 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.324602    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.377449 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.377416    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:02:52.381088 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.380958    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"]
Nov 03 22:02:52.388266 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.388239    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.388457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.388272    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.388574 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.388547    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfgz\" (UniqueName: \"kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.427108 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.427071    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:02:52.464059 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:02:52.464017    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:02:52.489231 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.489197    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.489231 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.489234    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.489496 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.489403    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfgz\" (UniqueName: \"kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.489873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.489852    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.492066 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.492037    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.558244 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.558211    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfgz\" (UniqueName: \"kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz\") pod \"maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.587270 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.587179    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:02:52.587270 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.587205    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" podUID="fe290523-3644-4f1c-8870-63743b82d309" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 22:02:52.661515 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.661479    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:52.795196 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.794737    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 22:02:52.795196 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:52.794802    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 22:02:53.059277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:53.059251    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"]
Nov 03 22:02:53.792807 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:53.792716    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Nov 03 22:02:53.792807 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:53.792774    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Nov 03 22:02:54.004621 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:54.004581    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw" event={"ID":"2d47b111-3036-44ce-b563-4b18d81b7b80","Type":"ContainerStarted","Data":"661f1755efdf197eac2e70ac1f37a1fe5c522b9440fc6c550ff2e8eb562cdc0e"}
Nov 03 22:02:54.005097 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:54.004677    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw" event={"ID":"2d47b111-3036-44ce-b563-4b18d81b7b80","Type":"ContainerStarted","Data":"c92ab0efb5821d9d18a375307020db6a8e080f936e1e0f6751798052b56c02b2"}
Nov 03 22:02:54.119252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:54.119194    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw" podStartSLOduration=2.119179344 podStartE2EDuration="2.119179344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:02:54.116461131 +0000 UTC m=+2845.100457711" watchObservedRunningTime="2026-11-03 22:02:54.119179344 +0000 UTC m=+2845.103175903"
Nov 03 22:02:55.016862 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:55.016823    5244 generic.go:356] "Generic (PLEG): container finished" podID="2d47b111-3036-44ce-b563-4b18d81b7b80" containerID="661f1755efdf197eac2e70ac1f37a1fe5c522b9440fc6c550ff2e8eb562cdc0e" exitCode=0
Nov 03 22:02:55.017431 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:55.016905    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw" event={"ID":"2d47b111-3036-44ce-b563-4b18d81b7b80","Type":"ContainerDied","Data":"661f1755efdf197eac2e70ac1f37a1fe5c522b9440fc6c550ff2e8eb562cdc0e"}
Nov 03 22:02:55.248722 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:55.248686    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:55.249031 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:55.248997    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:55.250034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:55.250015    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:02:56.208735 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.208702    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:56.349409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.348991    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca\") pod \"2d47b111-3036-44ce-b563-4b18d81b7b80\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") "
Nov 03 22:02:56.349409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.349070    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz\" (UniqueName: \"kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz\") pod \"2d47b111-3036-44ce-b563-4b18d81b7b80\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") "
Nov 03 22:02:56.349409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.349310    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca\") pod \"2d47b111-3036-44ce-b563-4b18d81b7b80\" (UID: \"2d47b111-3036-44ce-b563-4b18d81b7b80\") "
Nov 03 22:02:56.349723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.349437    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca" pod "2d47b111-3036-44ce-b563-4b18d81b7b80" (UID: "2d47b111-3036-44ce-b563-4b18d81b7b80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:02:56.362355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.349792    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:56.362355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.352308    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz" pod "2d47b111-3036-44ce-b563-4b18d81b7b80" (UID: "2d47b111-3036-44ce-b563-4b18d81b7b80"). InnerVolumeSpecName "kube-api-access-khfgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:02:56.362355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.352439    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca" pod "2d47b111-3036-44ce-b563-4b18d81b7b80" (UID: "2d47b111-3036-44ce-b563-4b18d81b7b80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:02:56.450898 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.450860    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d47b111-3036-44ce-b563-4b18d81b7b80-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:56.450898 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:56.450900    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khfgz\" (UniqueName: \"kubernetes.io/projected/2d47b111-3036-44ce-b563-4b18d81b7b80-kube-api-access-khfgz\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:02:57.048037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:57.048003    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"
Nov 03 22:02:57.048318 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:57.048278    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw" event={"ID":"2d47b111-3036-44ce-b563-4b18d81b7b80","Type":"ContainerDied","Data":"c92ab0efb5821d9d18a375307020db6a8e080f936e1e0f6751798052b56c02b2"}
Nov 03 22:02:57.048648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:02:57.048521    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c92ab0efb5821d9d18a375307020db6a8e080f936e1e0f6751798052b56c02b2"
Nov 03 22:03:01.792798 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:01.792716    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 22:03:01.925845 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:01.925814    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 22:03:01.952874 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:01.952848    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5\" (UniqueName: \"kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5\") pod \"4fda315c-f98f-4687-8c2e-e01f4f293891\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") "
Nov 03 22:03:01.953083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:01.952905    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume\") pod \"4fda315c-f98f-4687-8c2e-e01f4f293891\" (UID: \"4fda315c-f98f-4687-8c2e-e01f4f293891\") "
Nov 03 22:03:01.953312 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:01.953290    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume" pod "4fda315c-f98f-4687-8c2e-e01f4f293891" (UID: "4fda315c-f98f-4687-8c2e-e01f4f293891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:01.955112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:01.955090    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5" pod "4fda315c-f98f-4687-8c2e-e01f4f293891" (UID: "4fda315c-f98f-4687-8c2e-e01f4f293891"). InnerVolumeSpecName "kube-api-access-j9td5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:02.053773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.053690    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8\" (UniqueName: \"kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8\") pod \"fe290523-3644-4f1c-8870-63743b82d309\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") "
Nov 03 22:03:02.053947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.053812    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume\") pod \"fe290523-3644-4f1c-8870-63743b82d309\" (UID: \"fe290523-3644-4f1c-8870-63743b82d309\") "
Nov 03 22:03:02.054167 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.054139    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume" pod "fe290523-3644-4f1c-8870-63743b82d309" (UID: "fe290523-3644-4f1c-8870-63743b82d309"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:02.054307 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.054237    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9td5\" (UniqueName: \"kubernetes.io/projected/4fda315c-f98f-4687-8c2e-e01f4f293891-kube-api-access-j9td5\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:02.054307 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.054259    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fda315c-f98f-4687-8c2e-e01f4f293891-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:02.056192 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.056159    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8" pod "fe290523-3644-4f1c-8870-63743b82d309" (UID: "fe290523-3644-4f1c-8870-63743b82d309"). InnerVolumeSpecName "kube-api-access-px2v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:02.084289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.084242    5244 generic.go:356] "Generic (PLEG): container finished" podID="fe290523-3644-4f1c-8870-63743b82d309" containerID="55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1" exitCode=0
Nov 03 22:03:02.084289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.084290    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"
Nov 03 22:03:02.084590 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.084323    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" event={"ID":"fe290523-3644-4f1c-8870-63743b82d309","Type":"ContainerDied","Data":"55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1"}
Nov 03 22:03:02.084590 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.084438    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z" event={"ID":"fe290523-3644-4f1c-8870-63743b82d309","Type":"ContainerDied","Data":"be071f00d9f20df04f1d024171e136ecd823efcf108785d63883a75b977dfe33"}
Nov 03 22:03:02.084590 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.084467    5244 scope.go:122] "RemoveContainer" containerID="55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1"
Nov 03 22:03:02.086234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.086049    5244 generic.go:356] "Generic (PLEG): container finished" podID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerID="b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54" exitCode=0
Nov 03 22:03:02.086234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.086092    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"
Nov 03 22:03:02.086234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.086167    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" event={"ID":"4fda315c-f98f-4687-8c2e-e01f4f293891","Type":"ContainerDied","Data":"b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54"}
Nov 03 22:03:02.086234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.086207    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl" event={"ID":"4fda315c-f98f-4687-8c2e-e01f4f293891","Type":"ContainerDied","Data":"0a2cf742aba6f57459acb0229bd8152f8b4c7df225667951e532df41a079a5b7"}
Nov 03 22:03:02.096705 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.096579    5244 scope.go:122] "RemoveContainer" containerID="55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1"
Nov 03 22:03:02.097109 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:02.097084    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1\": container with ID starting with 55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1 not found: ID does not exist" containerID="55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1"
Nov 03 22:03:02.097216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.097120    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1"} err="failed to get container status \"55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1\": rpc error: code = NotFound desc = could not find container \"55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1\": container with ID starting with 55defe2c95682c2b1db2003b234b14e383e51735c20c174193bbb01a5988b6d1 not found: ID does not exist"
Nov 03 22:03:02.097216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.097137    5244 scope.go:122] "RemoveContainer" containerID="b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54"
Nov 03 22:03:02.109466 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.109438    5244 scope.go:122] "RemoveContainer" containerID="b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54"
Nov 03 22:03:02.109835 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:02.109810    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54\": container with ID starting with b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54 not found: ID does not exist" containerID="b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54"
Nov 03 22:03:02.109921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.109851    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54"} err="failed to get container status \"b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54\": rpc error: code = NotFound desc = could not find container \"b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54\": container with ID starting with b31babf1b2b1fdc01c9180c15844182d3e43ceecb958136e6e6a4c90dd6e8c54 not found: ID does not exist"
Nov 03 22:03:02.155772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.155742    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px2v8\" (UniqueName: \"kubernetes.io/projected/fe290523-3644-4f1c-8870-63743b82d309-kube-api-access-px2v8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:02.155772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.155773    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe290523-3644-4f1c-8870-63743b82d309-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:02.247525 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.247491    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"]
Nov 03 22:03:02.251586 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.251562    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb3520-8469b94854-2dpwl"]
Nov 03 22:03:02.359488 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.359390    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"]
Nov 03 22:03:02.362324 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:02.362303    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb3520-7f4b496f75-6jk2z"]
Nov 03 22:03:03.135922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:03.135892    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" path="/var/lib/kubelet/pods/4fda315c-f98f-4687-8c2e-e01f4f293891/volumes"
Nov 03 22:03:03.136344 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:03.136291    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe290523-3644-4f1c-8870-63743b82d309" path="/var/lib/kubelet/pods/fe290523-3644-4f1c-8870-63743b82d309/volumes"
Nov 03 22:03:04.373101 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:04.373067    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"]
Nov 03 22:03:04.373566 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:04.373270    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy" containerID="cri-o://f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7" gracePeriod=30
Nov 03 22:03:09.643577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.643509    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 22:03:09.670886 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.670852    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.670954    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.670985    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671009    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671027    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671046    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671078    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671144    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671219    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data\") pod \"5e519501-99b3-474a-b49d-71b43984f860\" (UID: \"5e519501-99b3-474a-b49d-71b43984f860\") "
Nov 03 22:03:09.671540 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671435    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:09.671540 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671465    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:09.671664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671541    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:09.671664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671632    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:09.671833 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671766    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:09.671833 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671815    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.671833 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671836    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.671833 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671845    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.671833 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.671854    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.673641 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.673615    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:03:09.673774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.673663    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:03:09.673906 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.673889    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:09.673967 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.673914    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx" pod "5e519501-99b3-474a-b49d-71b43984f860" (UID: "5e519501-99b3-474a-b49d-71b43984f860"). InnerVolumeSpecName "kube-api-access-v59cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:09.773307 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.773258    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519501-99b3-474a-b49d-71b43984f860-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.773307 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.773306    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59cx\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-kube-api-access-v59cx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.773563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.773323    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519501-99b3-474a-b49d-71b43984f860-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.773563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.773353    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519501-99b3-474a-b49d-71b43984f860-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:09.773563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:09.773365    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519501-99b3-474a-b49d-71b43984f860-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:10.142372 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.142318    5244 generic.go:356] "Generic (PLEG): container finished" podID="5e519501-99b3-474a-b49d-71b43984f860" containerID="f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7" exitCode=0
Nov 03 22:03:10.142372 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.142360    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" event={"ID":"5e519501-99b3-474a-b49d-71b43984f860","Type":"ContainerDied","Data":"f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7"}
Nov 03 22:03:10.142648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.142391    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"
Nov 03 22:03:10.142648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.142429    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn" event={"ID":"5e519501-99b3-474a-b49d-71b43984f860","Type":"ContainerDied","Data":"95d77c0ab5bd097e8ff998d147d3828a1932dd9695cde352c8022334ffa94e5c"}
Nov 03 22:03:10.142648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.142457    5244 scope.go:122] "RemoveContainer" containerID="f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7"
Nov 03 22:03:10.153513 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.153481    5244 scope.go:122] "RemoveContainer" containerID="f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7"
Nov 03 22:03:10.153800 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:10.153781    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7\": container with ID starting with f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7 not found: ID does not exist" containerID="f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7"
Nov 03 22:03:10.153851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.153810    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7"} err="failed to get container status \"f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7\": rpc error: code = NotFound desc = could not find container \"f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7\": container with ID starting with f607f72a060cffc38945b5b4fbae5fbe6cc854d357bc5f943bc1ea985ad8dff7 not found: ID does not exist"
Nov 03 22:03:10.305711 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.305675    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"]
Nov 03 22:03:10.355445 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:10.355404    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-87e1cf-openshift-default-7fdb9f46c9-pgpmn"]
Nov 03 22:03:11.136710 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:11.136678    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e519501-99b3-474a-b49d-71b43984f860" path="/var/lib/kubelet/pods/5e519501-99b3-474a-b49d-71b43984f860/volumes"
Nov 03 22:03:11.281962 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:11.281929    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"]
Nov 03 22:03:11.390947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:11.390802    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"]
Nov 03 22:03:16.282452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.282402    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7" podUID="49061d70-1848-4253-8a5f-c0899631d52e" containerName="payload-processing" containerID="cri-o://2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c" gracePeriod=25
Nov 03 22:03:16.391205 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.391146    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw" podUID="c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" containerName="payload-pre-processing" containerID="cri-o://e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64" gracePeriod=25
Nov 03 22:03:16.546744 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.546668    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 22:03:16.648169 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.648137    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7\" (UniqueName: \"kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7\") pod \"49061d70-1848-4253-8a5f-c0899631d52e\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") "
Nov 03 22:03:16.648299 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.648281    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume\") pod \"49061d70-1848-4253-8a5f-c0899631d52e\" (UID: \"49061d70-1848-4253-8a5f-c0899631d52e\") "
Nov 03 22:03:16.648708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.648682    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume" pod "49061d70-1848-4253-8a5f-c0899631d52e" (UID: "49061d70-1848-4253-8a5f-c0899631d52e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:16.650594 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.650562    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7" pod "49061d70-1848-4253-8a5f-c0899631d52e" (UID: "49061d70-1848-4253-8a5f-c0899631d52e"). InnerVolumeSpecName "kube-api-access-6p6m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:16.654723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.654702    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 22:03:16.749817 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.749773    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9\" (UniqueName: \"kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9\") pod \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") "
Nov 03 22:03:16.750005 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.749865    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume\") pod \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\" (UID: \"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d\") "
Nov 03 22:03:16.750119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.750103    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p6m7\" (UniqueName: \"kubernetes.io/projected/49061d70-1848-4253-8a5f-c0899631d52e-kube-api-access-6p6m7\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:16.750157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.750122    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49061d70-1848-4253-8a5f-c0899631d52e-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:16.750241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.750217    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume" pod "c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" (UID: "c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:03:16.752081 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.752059    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9" pod "c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" (UID: "c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d"). InnerVolumeSpecName "kube-api-access-6f5b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:03:16.851064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.850963    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5b9\" (UniqueName: \"kubernetes.io/projected/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-kube-api-access-6f5b9\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:16.851064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:16.851004    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:03:17.187343 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.187296    5244 generic.go:356] "Generic (PLEG): container finished" podID="49061d70-1848-4253-8a5f-c0899631d52e" containerID="2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c" exitCode=0
Nov 03 22:03:17.187522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.187359    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"
Nov 03 22:03:17.187522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.187389    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7" event={"ID":"49061d70-1848-4253-8a5f-c0899631d52e","Type":"ContainerDied","Data":"2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c"}
Nov 03 22:03:17.187522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.187450    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7" event={"ID":"49061d70-1848-4253-8a5f-c0899631d52e","Type":"ContainerDied","Data":"c9d2ec27339e8d079d551ad397f76516793cd6fd6e57153d086335b145070f99"}
Nov 03 22:03:17.187522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.187468    5244 scope.go:122] "RemoveContainer" containerID="2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c"
Nov 03 22:03:17.188994 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.188971    5244 generic.go:356] "Generic (PLEG): container finished" podID="c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" containerID="e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64" exitCode=0
Nov 03 22:03:17.189129 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.189014    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"
Nov 03 22:03:17.189250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.189226    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw" event={"ID":"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d","Type":"ContainerDied","Data":"e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64"}
Nov 03 22:03:17.189310 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.189282    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw" event={"ID":"c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d","Type":"ContainerDied","Data":"10c761ac07460ea0338a83032d723ce37b0320f7fde3b099dd77e33c8858b6bf"}
Nov 03 22:03:17.197392 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.197371    5244 scope.go:122] "RemoveContainer" containerID="2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c"
Nov 03 22:03:17.197713 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:17.197690    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c\": container with ID starting with 2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c not found: ID does not exist" containerID="2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c"
Nov 03 22:03:17.197772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.197721    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c"} err="failed to get container status \"2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c\": rpc error: code = NotFound desc = could not find container \"2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c\": container with ID starting with 2a912ff2980e886e276b4828d3048f70afba41e3005a097dba060575492bbc9c not found: ID does not exist"
Nov 03 22:03:17.197772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.197739    5244 scope.go:122] "RemoveContainer" containerID="e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64"
Nov 03 22:03:17.206850 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.206824    5244 scope.go:122] "RemoveContainer" containerID="e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64"
Nov 03 22:03:17.207136 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:17.207118    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64\": container with ID starting with e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64 not found: ID does not exist" containerID="e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64"
Nov 03 22:03:17.207187 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.207143    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64"} err="failed to get container status \"e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64\": rpc error: code = NotFound desc = could not find container \"e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64\": container with ID starting with e4b5e05e0532e468f0169098ca9d94c546067cca15daca81e02eda6236fd8a64 not found: ID does not exist"
Nov 03 22:03:17.351027 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.350994    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"]
Nov 03 22:03:17.353966 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.353938    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4644ae-777b8cb4f6-j8pn7"]
Nov 03 22:03:17.469965 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.469910    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"]
Nov 03 22:03:17.474870 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:17.474830    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4644ae-799dbbcbc-df4nw"]
Nov 03 22:03:18.078817 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.078786    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"]
Nov 03 22:03:18.079204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079192    5244 cpu_manager.go:414] "containerMap: removing container" podUID="49061d70-1848-4253-8a5f-c0899631d52e" containerName="payload-processing"
Nov 03 22:03:18.079250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079205    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49061d70-1848-4253-8a5f-c0899631d52e" containerName="payload-processing"
Nov 03 22:03:18.079250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079224    5244 cpu_manager.go:414] "containerMap: removing container" podUID="fe290523-3644-4f1c-8870-63743b82d309" containerName="payload-pre-processing"
Nov 03 22:03:18.079250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079229    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe290523-3644-4f1c-8870-63743b82d309" containerName="payload-pre-processing"
Nov 03 22:03:18.079250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079242    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2d47b111-3036-44ce-b563-4b18d81b7b80" containerName="revoke-keys"
Nov 03 22:03:18.079250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079248    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d47b111-3036-44ce-b563-4b18d81b7b80" containerName="revoke-keys"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079262    5244 cpu_manager.go:414] "containerMap: removing container" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerName="payload-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079267    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerName="payload-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079282    5244 cpu_manager.go:414] "containerMap: removing container" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079287    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079294    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" containerName="payload-pre-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079299    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" containerName="payload-pre-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079395    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fda315c-f98f-4687-8c2e-e01f4f293891" containerName="payload-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079410    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e519501-99b3-474a-b49d-71b43984f860" containerName="istio-proxy"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079416    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" containerName="payload-pre-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079423    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="49061d70-1848-4253-8a5f-c0899631d52e" containerName="payload-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079430    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe290523-3644-4f1c-8870-63743b82d309" containerName="payload-pre-processing"
Nov 03 22:03:18.079437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.079437    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d47b111-3036-44ce-b563-4b18d81b7b80" containerName="revoke-keys"
Nov 03 22:03:18.084201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.084176    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.136904 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.136869    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-a49a32\""
Nov 03 22:03:18.137094 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.136926    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-a49a32-dockercfg-wzs2t\""
Nov 03 22:03:18.139921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.139894    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"]
Nov 03 22:03:18.191355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.188118    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cpx9\" (UniqueName: \"kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9\") pod \"payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.191355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.188224    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.259993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.259954    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"]
Nov 03 22:03:18.263944 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.263925    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.289707 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.289672    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.289960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.289913    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.290181 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.290078    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cst4\" (UniqueName: \"kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4\") pod \"payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.290242 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.290181    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cpx9\" (UniqueName: \"kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9\") pod \"payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.290553 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.290528    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.319942 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.319911    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"]
Nov 03 22:03:18.347353 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.347246    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cpx9\" (UniqueName: \"kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9\") pod \"payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.391338 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.391292    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.391729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.391374    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cst4\" (UniqueName: \"kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4\") pod \"payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.392019 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.391999    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.397520 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.397492    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:18.449291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.449260    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cst4\" (UniqueName: \"kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4\") pod \"payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.577814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.577776    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:18.776207 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.776183    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"]
Nov 03 22:03:18.778391 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:03:18.778361    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6da5f32_0288_4e77_8599_deaaab3ca3f6.slice/crio-8bfa55c572154440f9483e16f79beb94aaf2d96746e24e4814364fc107a351e0 WatchSource:0}: Error finding container 8bfa55c572154440f9483e16f79beb94aaf2d96746e24e4814364fc107a351e0: Status 404 returned error can't find the container with id 8bfa55c572154440f9483e16f79beb94aaf2d96746e24e4814364fc107a351e0
Nov 03 22:03:18.952897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:18.952865    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"]
Nov 03 22:03:18.953828 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:03:18.953794    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c4b706_d80c_41e7_81b0_e51025177750.slice/crio-67ea267a5ff02e9af306887daf0f584431259c6e47e209d9c7bb6d0ebc7d376d WatchSource:0}: Error finding container 67ea267a5ff02e9af306887daf0f584431259c6e47e209d9c7bb6d0ebc7d376d: Status 404 returned error can't find the container with id 67ea267a5ff02e9af306887daf0f584431259c6e47e209d9c7bb6d0ebc7d376d
Nov 03 22:03:19.135918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.135888    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49061d70-1848-4253-8a5f-c0899631d52e" path="/var/lib/kubelet/pods/49061d70-1848-4253-8a5f-c0899631d52e/volumes"
Nov 03 22:03:19.136297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.136273    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d" path="/var/lib/kubelet/pods/c2adf496-a1d9-4dd5-8ce0-cc6c3131c74d/volumes"
Nov 03 22:03:19.243303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.243198    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j" event={"ID":"49c4b706-d80c-41e7-81b0-e51025177750","Type":"ContainerStarted","Data":"3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81"}
Nov 03 22:03:19.243303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.243259    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j" event={"ID":"49c4b706-d80c-41e7-81b0-e51025177750","Type":"ContainerStarted","Data":"67ea267a5ff02e9af306887daf0f584431259c6e47e209d9c7bb6d0ebc7d376d"}
Nov 03 22:03:19.243562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.243395    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:19.245004 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.244982    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm" event={"ID":"c6da5f32-0288-4e77-8599-deaaab3ca3f6","Type":"ContainerStarted","Data":"1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615"}
Nov 03 22:03:19.245133 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.245019    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm" event={"ID":"c6da5f32-0288-4e77-8599-deaaab3ca3f6","Type":"ContainerStarted","Data":"8bfa55c572154440f9483e16f79beb94aaf2d96746e24e4814364fc107a351e0"}
Nov 03 22:03:19.245133 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.245072    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:19.406477 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:19.406425    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j" podStartSLOduration=1.406409101 podStartE2EDuration="1.406409101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:19.404838809 +0000 UTC m=+2870.388835388" watchObservedRunningTime="2026-11-03 22:03:19.406409101 +0000 UTC m=+2870.390405660"
Nov 03 22:03:22.436944 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:22.436903    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:03:23.831293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:23.831230    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm" podStartSLOduration=5.831212889 podStartE2EDuration="5.831212889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:19.534282811 +0000 UTC m=+2870.518279406" watchObservedRunningTime="2026-11-03 22:03:23.831212889 +0000 UTC m=+2874.815209447"
Nov 03 22:03:23.832771 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:23.832750    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:03:29.959849 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:29.959820    5244 scope.go:122] "RemoveContainer" containerID="e6db2ae4acdbe3dc1240a916407483d1f02b0956ea6de4ace3deb0a7664512f8"
Nov 03 22:03:29.969916 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:29.969893    5244 scope.go:122] "RemoveContainer" containerID="c1a822cde5ba11130f1174a7ff3942c2803c58c33d9a78817ec586b624c7bb44"
Nov 03 22:03:30.254640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:30.254556    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:03:30.254781 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:30.254750    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:03:33.518443 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:33.518405    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"]
Nov 03 22:03:33.567410 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:33.567374    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4644ae-z56dw"]
Nov 03 22:03:35.135990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:35.135956    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d47b111-3036-44ce-b563-4b18d81b7b80" path="/var/lib/kubelet/pods/2d47b111-3036-44ce-b563-4b18d81b7b80/volumes"
Nov 03 22:03:42.797712 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.797629    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"]
Nov 03 22:03:42.803273 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.803248    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:42.906273 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.906219    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-29dbd9-serving-cert\""
Nov 03 22:03:42.909296 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.909269    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"]
Nov 03 22:03:42.988977 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.988947    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:42.989157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.989013    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:42.989157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.989124    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmtp\" (UniqueName: \"kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:42.989250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:42.989229    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.089910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.089821    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.089910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.089894    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmtp\" (UniqueName: \"kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.090172 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.089987    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.090172 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.090070    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.090172 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:43.090119    5244 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-serving-cert: secret "maas-api-e2e-shared-a-w4-29dbd9-serving-cert" not found
Nov 03 22:03:43.090320 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:43.090214    5244 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls podName:2dfbf5b6-d4b6-4216-b674-336324640a2d nodeName:}" failed. No retries permitted until 2026-11-03 22:03:43.590191048 +0000 UTC m=+2894.574187599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls") pod "maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" (UID: "2dfbf5b6-d4b6-4216-b674-336324640a2d") : secret "maas-api-e2e-shared-a-w4-29dbd9-serving-cert" not found
Nov 03 22:03:43.090813 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.090794    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.092642 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.092621    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.147041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.147012    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmtp\" (UniqueName: \"kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.609145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.609107    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.611699 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.611671    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:43.716277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:43.716234    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:44.095640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:44.095608    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"]
Nov 03 22:03:44.096873 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:03:44.096843    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfbf5b6_d4b6_4216_b674_336324640a2d.slice/crio-342f47c558df98106e7de93653cc4122d6be98c5786b4232720a16821bc59611 WatchSource:0}: Error finding container 342f47c558df98106e7de93653cc4122d6be98c5786b4232720a16821bc59611: Status 404 returned error can't find the container with id 342f47c558df98106e7de93653cc4122d6be98c5786b4232720a16821bc59611
Nov 03 22:03:44.098295 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:44.098279    5244 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 22:03:44.405061 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:44.404970    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" event={"ID":"2dfbf5b6-d4b6-4216-b674-336324640a2d","Type":"ContainerStarted","Data":"342f47c558df98106e7de93653cc4122d6be98c5786b4232720a16821bc59611"}
Nov 03 22:03:45.412466 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:45.412379    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" event={"ID":"2dfbf5b6-d4b6-4216-b674-336324640a2d","Type":"ContainerStarted","Data":"f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77"}
Nov 03 22:03:45.412859 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:45.412569    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:45.530082 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:45.530022    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" podStartSLOduration=2.6699152550000003 podStartE2EDuration="3.53000746s" totalImagesPullingTime="860.092205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:03:45.525776228 +0000 UTC m=+2896.509772804" watchObservedRunningTime="2026-11-03 22:03:45.53000746 +0000 UTC m=+2896.514004056"
Nov 03 22:03:50.036485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:50.036453    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"]
Nov 03 22:03:50.084839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:50.084808    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb3520-lls9z"]
Nov 03 22:03:51.136881 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:51.136802    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da54f1cb-2cb4-4c8e-a84f-b939af7b9359" path="/var/lib/kubelet/pods/da54f1cb-2cb4-4c8e-a84f-b939af7b9359/volumes"
Nov 03 22:03:51.146695 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:51.146656    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"]
Nov 03 22:03:51.196105 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:51.196069    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7990bd-nn4wz"]
Nov 03 22:03:51.424244 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:51.424174    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:03:52.437911 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:03:52.437875    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:03:53.136103 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:53.136071    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebedbc8-30c8-4059-a5d9-f4f94e7f830e" path="/var/lib/kubelet/pods/5ebedbc8-30c8-4059-a5d9-f4f94e7f830e/volumes"
Nov 03 22:03:57.907355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:57.905868    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"]
Nov 03 22:03:58.013086 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:03:58.013048    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"]
Nov 03 22:04:02.906921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:02.906865    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4" podUID="e56564cc-8866-42de-9dc0-4923ee287e65" containerName="payload-processing" containerID="cri-o://b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075" gracePeriod=25
Nov 03 22:04:03.013868 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.013825    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6" podUID="fa95d5b1-c1ca-4a60-9695-6f151886aeb8" containerName="payload-pre-processing" containerID="cri-o://324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6" gracePeriod=25
Nov 03 22:04:03.217768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.217733    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:04:03.301043 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.301014    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:04:03.306309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.306284    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5\" (UniqueName: \"kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5\") pod \"e56564cc-8866-42de-9dc0-4923ee287e65\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") "
Nov 03 22:04:03.306467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.306410    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume\") pod \"e56564cc-8866-42de-9dc0-4923ee287e65\" (UID: \"e56564cc-8866-42de-9dc0-4923ee287e65\") "
Nov 03 22:04:03.306774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.306749    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume" pod "e56564cc-8866-42de-9dc0-4923ee287e65" (UID: "e56564cc-8866-42de-9dc0-4923ee287e65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:03.308737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.308717    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5" pod "e56564cc-8866-42de-9dc0-4923ee287e65" (UID: "e56564cc-8866-42de-9dc0-4923ee287e65"). InnerVolumeSpecName "kube-api-access-6mxs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:03.407877 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.407845    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx\" (UniqueName: \"kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx\") pod \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") "
Nov 03 22:04:03.408094 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.407894    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume\") pod \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\" (UID: \"fa95d5b1-c1ca-4a60-9695-6f151886aeb8\") "
Nov 03 22:04:03.408174 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.408128    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mxs5\" (UniqueName: \"kubernetes.io/projected/e56564cc-8866-42de-9dc0-4923ee287e65-kube-api-access-6mxs5\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.408174 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.408141    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56564cc-8866-42de-9dc0-4923ee287e65-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.408293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.408270    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume" pod "fa95d5b1-c1ca-4a60-9695-6f151886aeb8" (UID: "fa95d5b1-c1ca-4a60-9695-6f151886aeb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:04:03.410193 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.410165    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx" pod "fa95d5b1-c1ca-4a60-9695-6f151886aeb8" (UID: "fa95d5b1-c1ca-4a60-9695-6f151886aeb8"). InnerVolumeSpecName "kube-api-access-bqzrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:04:03.511353 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.510564    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.511353 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.510602    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzrx\" (UniqueName: \"kubernetes.io/projected/fa95d5b1-c1ca-4a60-9695-6f151886aeb8-kube-api-access-bqzrx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:04:03.613975 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.613941    5244 generic.go:356] "Generic (PLEG): container finished" podID="fa95d5b1-c1ca-4a60-9695-6f151886aeb8" containerID="324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6" exitCode=0
Nov 03 22:04:03.614161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.613994    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"
Nov 03 22:04:03.614161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.614018    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6" event={"ID":"fa95d5b1-c1ca-4a60-9695-6f151886aeb8","Type":"ContainerDied","Data":"324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6"}
Nov 03 22:04:03.614161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.614075    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6" event={"ID":"fa95d5b1-c1ca-4a60-9695-6f151886aeb8","Type":"ContainerDied","Data":"c04d46b94771d6ccb709def5df038becf3394e778cc6a1da75216e56f92a5afb"}
Nov 03 22:04:03.614161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.614106    5244 scope.go:122] "RemoveContainer" containerID="324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6"
Nov 03 22:04:03.615671 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.615647    5244 generic.go:356] "Generic (PLEG): container finished" podID="e56564cc-8866-42de-9dc0-4923ee287e65" containerID="b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075" exitCode=0
Nov 03 22:04:03.615800 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.615691    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4" event={"ID":"e56564cc-8866-42de-9dc0-4923ee287e65","Type":"ContainerDied","Data":"b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075"}
Nov 03 22:04:03.615800 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.615754    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4" event={"ID":"e56564cc-8866-42de-9dc0-4923ee287e65","Type":"ContainerDied","Data":"239602d7e85472654cba564e34542143c90a7d31219be22ff88e100a86788cfa"}
Nov 03 22:04:03.615800 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.615701    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"
Nov 03 22:04:03.629454 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.629423    5244 scope.go:122] "RemoveContainer" containerID="324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6"
Nov 03 22:04:03.630000 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:04:03.629974    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6\": container with ID starting with 324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6 not found: ID does not exist" containerID="324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6"
Nov 03 22:04:03.630093 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.630011    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6"} err="failed to get container status \"324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6\": rpc error: code = NotFound desc = could not find container \"324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6\": container with ID starting with 324c0119653c961dec184372c7a81b34e15c2129e4e1cd5e40ac88e22bb5a4f6 not found: ID does not exist"
Nov 03 22:04:03.630093 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.630038    5244 scope.go:122] "RemoveContainer" containerID="b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075"
Nov 03 22:04:03.641459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.641426    5244 scope.go:122] "RemoveContainer" containerID="b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075"
Nov 03 22:04:03.641746 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:04:03.641725    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075\": container with ID starting with b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075 not found: ID does not exist" containerID="b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075"
Nov 03 22:04:03.641800 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.641754    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075"} err="failed to get container status \"b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075\": rpc error: code = NotFound desc = could not find container \"b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075\": container with ID starting with b8f060b50ada95712229b61c837ea5ae990bbd9a11f37789937b69b8a6fa7075 not found: ID does not exist"
Nov 03 22:04:03.779255 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.779171    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"]
Nov 03 22:04:03.784598 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.784573    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bd35c-7f6d457db6-pcnf4"]
Nov 03 22:04:03.892044 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.892004    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"]
Nov 03 22:04:03.897431 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:03.897398    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bd35c-f74959d95-d9kk6"]
Nov 03 22:04:05.136768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:05.136735    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56564cc-8866-42de-9dc0-4923ee287e65" path="/var/lib/kubelet/pods/e56564cc-8866-42de-9dc0-4923ee287e65/volumes"
Nov 03 22:04:05.137185 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:05.137168    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa95d5b1-c1ca-4a60-9695-6f151886aeb8" path="/var/lib/kubelet/pods/fa95d5b1-c1ca-4a60-9695-6f151886aeb8/volumes"
Nov 03 22:04:09.269407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.269368    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"]
Nov 03 22:04:09.270038 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.270015    5244 cpu_manager.go:414] "containerMap: removing container" podUID="fa95d5b1-c1ca-4a60-9695-6f151886aeb8" containerName="payload-pre-processing"
Nov 03 22:04:09.270133 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.270040    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa95d5b1-c1ca-4a60-9695-6f151886aeb8" containerName="payload-pre-processing"
Nov 03 22:04:09.270133 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.270056    5244 cpu_manager.go:414] "containerMap: removing container" podUID="e56564cc-8866-42de-9dc0-4923ee287e65" containerName="payload-processing"
Nov 03 22:04:09.270133 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.270063    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56564cc-8866-42de-9dc0-4923ee287e65" containerName="payload-processing"
Nov 03 22:04:09.270286 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.270221    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa95d5b1-c1ca-4a60-9695-6f151886aeb8" containerName="payload-pre-processing"
Nov 03 22:04:09.270286 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.270236    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56564cc-8866-42de-9dc0-4923ee287e65" containerName="payload-processing"
Nov 03 22:04:09.275185 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.275162    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.330085 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.330051    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"]
Nov 03 22:04:09.391355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.379673    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e15bbc-dockercfg-7q647\""
Nov 03 22:04:09.391355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.379818    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e15bbc\""
Nov 03 22:04:09.400317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.400268    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcgf\" (UniqueName: \"kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf\") pod \"payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.400738 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.400713    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.435832 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.435661    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"]
Nov 03 22:04:09.460018 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.459989    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.480298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.480263    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"]
Nov 03 22:04:09.502510 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.502478    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.502703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.502534    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcgf\" (UniqueName: \"kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf\") pod \"payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.502703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.502607    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckx4\" (UniqueName: \"kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4\") pod \"payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.502846 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.502749    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.503550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.503527    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.559873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.559797    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcgf\" (UniqueName: \"kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf\") pod \"payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.589136 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.589093    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:09.603780 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.603753    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.603867 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.603833    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckx4\" (UniqueName: \"kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4\") pod \"payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.604519 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.604497    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.662696 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.662630    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckx4\" (UniqueName: \"kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4\") pod \"payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.778466 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.778435    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:09.968355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:09.968307    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"]
Nov 03 22:04:09.970509 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:04:09.970479    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221a1e8e_d593_4e40_b27a_b201d76c7e46.slice/crio-f2e2affb997709162c608aab662c464868d532f6203b827f7a9409896705d721 WatchSource:0}: Error finding container f2e2affb997709162c608aab662c464868d532f6203b827f7a9409896705d721: Status 404 returned error can't find the container with id f2e2affb997709162c608aab662c464868d532f6203b827f7a9409896705d721
Nov 03 22:04:10.153343 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.153298    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"]
Nov 03 22:04:10.155448 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:04:10.155409    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81d8fbf_b920_45cd_94b7_d7f97aebaa97.slice/crio-32dea064bba0a4bf4816b2ed7bac36d96198f03b3fa19f868bee033258479e2a WatchSource:0}: Error finding container 32dea064bba0a4bf4816b2ed7bac36d96198f03b3fa19f868bee033258479e2a: Status 404 returned error can't find the container with id 32dea064bba0a4bf4816b2ed7bac36d96198f03b3fa19f868bee033258479e2a
Nov 03 22:04:10.686041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.685999    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c" event={"ID":"221a1e8e-d593-4e40-b27a-b201d76c7e46","Type":"ContainerStarted","Data":"d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d"}
Nov 03 22:04:10.686947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.686220    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:10.686947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.686267    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c" event={"ID":"221a1e8e-d593-4e40-b27a-b201d76c7e46","Type":"ContainerStarted","Data":"f2e2affb997709162c608aab662c464868d532f6203b827f7a9409896705d721"}
Nov 03 22:04:10.687900 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.687879    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf" event={"ID":"c81d8fbf-b920-45cd-94b7-d7f97aebaa97","Type":"ContainerStarted","Data":"1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca"}
Nov 03 22:04:10.688013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.687913    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf" event={"ID":"c81d8fbf-b920-45cd-94b7-d7f97aebaa97","Type":"ContainerStarted","Data":"32dea064bba0a4bf4816b2ed7bac36d96198f03b3fa19f868bee033258479e2a"}
Nov 03 22:04:10.688013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.687942    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:10.802214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.802158    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c" podStartSLOduration=1.802144995 podStartE2EDuration="1.802144995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:10.800005346 +0000 UTC m=+2921.784001906" watchObservedRunningTime="2026-11-03 22:04:10.802144995 +0000 UTC m=+2921.786141587"
Nov 03 22:04:10.915080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:10.915030    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf" podStartSLOduration=1.915016826 podStartE2EDuration="1.915016826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:04:10.912755983 +0000 UTC m=+2921.896752543" watchObservedRunningTime="2026-11-03 22:04:10.915016826 +0000 UTC m=+2921.899013392"
Nov 03 22:04:15.018234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:15.018192    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:04:21.698092 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:21.698064    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:04:21.698674 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:21.698129    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:04:22.437525 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:04:22.437484    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:04:23.044469 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:23.044435    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"]
Nov 03 22:04:23.094161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:23.094126    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b768a57a-cbpv7"]
Nov 03 22:04:23.137703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:23.137658    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93a5403-498b-4431-b5bc-ee35b291f889" path="/var/lib/kubelet/pods/d93a5403-498b-4431-b5bc-ee35b291f889/volumes"
Nov 03 22:04:25.804481 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:25.804449    5244 generic.go:356] "Generic (PLEG): container finished" podID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerID="41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb" exitCode=0
Nov 03 22:04:25.804960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:25.804530    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" event={"ID":"0ca5f268-88b9-4c99-a0e8-fcf382698cab","Type":"ContainerDied","Data":"41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"}
Nov 03 22:04:25.804960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:25.804954    5244 scope.go:122] "RemoveContainer" containerID="41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"
Nov 03 22:04:25.806547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:25.806513    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:04:25.806647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:25.806588    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:04:25.806647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:25.806635    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:04:26.812779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:26.812741    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" event={"ID":"0ca5f268-88b9-4c99-a0e8-fcf382698cab","Type":"ContainerStarted","Data":"2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e"}
Nov 03 22:04:27.588005 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:27.587969    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:04:27.589077 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:27.589049    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:04:27.589221 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:27.589097    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:04:28.587768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:28.587734    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:04:28.588202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:28.587791    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:04:28.891259 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:28.891181    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:04:29.587961 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:29.587917    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:04:29.588448 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:29.587980    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:04:30.587831 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:30.587788    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:04:30.588038 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:30.587851    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:04:31.588279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:31.588238    5244 patch_prober.go:29] interesting pod/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 22:04:31.588742 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:31.588319    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 22:04:32.595087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:32.595060    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:04:32.596099 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:32.596078    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:04:40.044789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:40.044756    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"]
Nov 03 22:04:40.093972 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:40.093942    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-368f0084-7kqwc"]
Nov 03 22:04:41.136270 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:41.136225    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563b243d-b8aa-49a7-baf8-7ec92852cb23" path="/var/lib/kubelet/pods/563b243d-b8aa-49a7-baf8-7ec92852cb23/volumes"
Nov 03 22:04:52.437063 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:04:52.437028    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:04:56.045844 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:56.045804    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"]
Nov 03 22:04:56.094356 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:56.094294    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0ff440-lrfhd"]
Nov 03 22:04:57.136791 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:04:57.136750    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb35f2d9-5133-4a13-be4c-f35bf89c4d71" path="/var/lib/kubelet/pods/cb35f2d9-5133-4a13-be4c-f35bf89c4d71/volumes"
Nov 03 22:05:14.017647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:14.017563    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:05:22.436647 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:05:22.436609    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:05:23.044783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:23.044748    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"]
Nov 03 22:05:23.093926 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:23.093886    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758cff82-gfmht"]
Nov 03 22:05:23.136145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:23.136107    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96415ea-7da2-4c38-9ff3-557bcbb71989" path="/var/lib/kubelet/pods/b96415ea-7da2-4c38-9ff3-557bcbb71989/volumes"
Nov 03 22:05:30.073577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:30.073544    5244 scope.go:122] "RemoveContainer" containerID="92857abca0a7e55de7a4f88f9d53ad3e606939e1866bd9cbdb6edb9b7cab734d"
Nov 03 22:05:40.045612 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:40.045575    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"]
Nov 03 22:05:40.094554 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:40.094519    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58d7e957-hbxbc"]
Nov 03 22:05:41.150183 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.150149    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055b8496-b4ed-4c0b-9367-c95ba5d4b230" path="/var/lib/kubelet/pods/055b8496-b4ed-4c0b-9367-c95ba5d4b230/volumes"
Nov 03 22:05:41.554356 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.546443    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"]
Nov 03 22:05:41.572438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.564308    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.614435 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.614399    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"]
Nov 03 22:05:41.627100 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.627063    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-27a3c3\""
Nov 03 22:05:41.660573 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.660526    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-27a3c3-dockercfg-8tkdn\""
Nov 03 22:05:41.729173 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.729136    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume\") pod \"payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") " pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.729413 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.729296    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8872\" (UniqueName: \"kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872\") pod \"payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") " pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.830683 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.830591    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8872\" (UniqueName: \"kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872\") pod \"payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") " pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.830860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.830736    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume\") pod \"payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") " pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.831465 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.831447    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume\") pod \"payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") " pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.887738 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.887704    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8872\" (UniqueName: \"kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872\") pod \"payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") " pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:41.897717 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:41.897690    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:42.279473 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:42.279438    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"]
Nov 03 22:05:42.286863 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:05:42.286828    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615e25f0_8693_4288_ad4f_b3b3c1b43608.slice/crio-d54729df8213471e906143f566d490598c588a1bca527ea401e45f05e9f4f74e WatchSource:0}: Error finding container d54729df8213471e906143f566d490598c588a1bca527ea401e45f05e9f4f74e: Status 404 returned error can't find the container with id d54729df8213471e906143f566d490598c588a1bca527ea401e45f05e9f4f74e
Nov 03 22:05:42.412830 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:42.412795    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8" event={"ID":"615e25f0-8693-4288-ad4f-b3b3c1b43608","Type":"ContainerStarted","Data":"13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77"}
Nov 03 22:05:42.413014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:42.412851    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8" event={"ID":"615e25f0-8693-4288-ad4f-b3b3c1b43608","Type":"ContainerStarted","Data":"d54729df8213471e906143f566d490598c588a1bca527ea401e45f05e9f4f74e"}
Nov 03 22:05:42.413014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:42.412998    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:05:42.527893 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:42.527836    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8" podStartSLOduration=1.527822677 podStartE2EDuration="1.527822677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:05:42.524583689 +0000 UTC m=+3013.508580248" watchObservedRunningTime="2026-11-03 22:05:42.527822677 +0000 UTC m=+3013.511819265"
Nov 03 22:05:42.580534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:42.580450    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:05:46.045883 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:46.045843    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"]
Nov 03 22:05:46.095492 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:46.095456    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eca68c3-td6zz"]
Nov 03 22:05:47.181093 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:47.181057    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b794cf-2d34-4536-b3c5-a3017371228c" path="/var/lib/kubelet/pods/f2b794cf-2d34-4536-b3c5-a3017371228c/volumes"
Nov 03 22:05:52.436506 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:05:52.436463    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:05:53.423772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:05:53.423744    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:06:00.045485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:00.045444    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"]
Nov 03 22:06:00.094768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:00.094732    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68850315-dlvj2"]
Nov 03 22:06:01.136751 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:01.136706    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93435a2d-d10a-4910-9e6f-13e79470b181" path="/var/lib/kubelet/pods/93435a2d-d10a-4910-9e6f-13e79470b181/volumes"
Nov 03 22:06:07.402620 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.402578    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"]
Nov 03 22:06:07.406597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.406577    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.460482 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.460444    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7fb477-openshift-default-dockercfg-f5m7z\""
Nov 03 22:06:07.524227 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.521221    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"]
Nov 03 22:06:07.586386 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.573974    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.590354 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.586706    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ffl\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.612354 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.590669    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.613052 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.613016    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.613374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.613351    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.613534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.613513    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.613668 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.613650    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.613913 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.613897    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.614037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.614018    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715022 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.714984    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ffl\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715038    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715093    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715128    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715153    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715181    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715206    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715231    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715360    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715616    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715691    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715880 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715719    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.715926 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.715880    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.716194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.716168    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.717874 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.717853    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.718096 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.718073    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.775123 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.775089    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:07.775318 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:07.775294    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ffl\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl\") pod \"e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") " pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:08.020978 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.020887    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:08.501936 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.501906    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"]
Nov 03 22:06:08.504378 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:06:08.504314    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17edd6f0_d922_46ea_8ad7_428466cfdab9.slice/crio-7cd67e897598a8b6affd341683e43296973d8bd4fc536ad79397cd39406579a8 WatchSource:0}: Error finding container 7cd67e897598a8b6affd341683e43296973d8bd4fc536ad79397cd39406579a8: Status 404 returned error can't find the container with id 7cd67e897598a8b6affd341683e43296973d8bd4fc536ad79397cd39406579a8
Nov 03 22:06:08.507590 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.507547    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:06:08.507687 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.507646    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:06:08.507729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.507694    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:06:08.646898 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.646853    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" event={"ID":"17edd6f0-d922-46ea-8ad7-428466cfdab9","Type":"ContainerStarted","Data":"cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"}
Nov 03 22:06:08.647087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.646927    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" event={"ID":"17edd6f0-d922-46ea-8ad7-428466cfdab9","Type":"ContainerStarted","Data":"7cd67e897598a8b6affd341683e43296973d8bd4fc536ad79397cd39406579a8"}
Nov 03 22:06:08.762985 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:08.762863    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podStartSLOduration=1.762846903 podStartE2EDuration="1.762846903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:08.762223815 +0000 UTC m=+3039.746220375" watchObservedRunningTime="2026-11-03 22:06:08.762846903 +0000 UTC m=+3039.746843464"
Nov 03 22:06:09.032837 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:09.031892    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:09.035704 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:09.035668    5244 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 22:06:09.035879 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:09.035727    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 22:06:10.026317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:10.026285    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:10.659664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:10.659627    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:10.660720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:10.660698    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:06:14.046174 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:14.046133    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"]
Nov 03 22:06:14.095583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:14.095548    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92618d08-7hdf4"]
Nov 03 22:06:15.138147 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:15.138114    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41dcbb8d-c1bf-4ce5-bf83-757700a117b9" path="/var/lib/kubelet/pods/41dcbb8d-c1bf-4ce5-bf83-757700a117b9/volumes"
Nov 03 22:06:22.437532 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:06:22.437487    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:06:25.952477 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:25.952434    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:06:30.044254 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:30.044215    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"]
Nov 03 22:06:30.093806 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:30.093772    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6072b12-f47s2"]
Nov 03 22:06:30.100607 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:30.100586    5244 scope.go:122] "RemoveContainer" containerID="4373cbc9ad6edd5d751a7317b7556fdfd2fade9a1bf64e6c7e83b5c0a890c0f0"
Nov 03 22:06:30.110689 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:30.110669    5244 scope.go:122] "RemoveContainer" containerID="bdd7755693bfd7ae02b6286a3bdd302f06391a6aeb4626e16ade8d1ea79b9d53"
Nov 03 22:06:30.119408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:30.119383    5244 scope.go:122] "RemoveContainer" containerID="7d32c040d9f3525eff776d244b0b0d17d4a01d69ea421ed35149d9dd9b9eef44"
Nov 03 22:06:31.142025 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:31.141990    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a0e87b-56f3-4997-b9c5-372034c836ea" path="/var/lib/kubelet/pods/b3a0e87b-56f3-4997-b9c5-372034c836ea/volumes"
Nov 03 22:06:36.044473 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:36.044432    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"]
Nov 03 22:06:36.093489 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:36.093460    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c2d9624-7skzd"]
Nov 03 22:06:37.136784 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:37.136704    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5e1231-6905-4b9f-a7f1-f5876dbc7926" path="/var/lib/kubelet/pods/1b5e1231-6905-4b9f-a7f1-f5876dbc7926/volumes"
Nov 03 22:06:46.045894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:46.045852    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"]
Nov 03 22:06:46.095183 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:46.095148    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b90f921-wxnsz"]
Nov 03 22:06:47.135882 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:47.135849    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623e22a1-7707-493b-90fc-662e7ae15f9d" path="/var/lib/kubelet/pods/623e22a1-7707-493b-90fc-662e7ae15f9d/volumes"
Nov 03 22:06:51.044320 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:51.044280    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"]
Nov 03 22:06:51.096628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:51.096586    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba5fb38d-cv62c"]
Nov 03 22:06:51.136229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:06:51.136151    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1" path="/var/lib/kubelet/pods/e6c1cc68-ebea-4c36-b1f3-afc0b02c12e1/volumes"
Nov 03 22:06:52.438208 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:06:52.438170    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:07:01.012396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:01.012362    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:07:08.044988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:08.044942    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"]
Nov 03 22:07:08.094411 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:08.094363    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68f3ba8-stkd8"]
Nov 03 22:07:09.136977 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:09.136946    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1044aafb-0b47-4c51-9dba-467b8d960124" path="/var/lib/kubelet/pods/1044aafb-0b47-4c51-9dba-467b8d960124/volumes"
Nov 03 22:07:13.634360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.634310    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"]
Nov 03 22:07:13.638470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.638449    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.708568 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.708538    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-29dbd9\"/\"kube-root-ca.crt\""
Nov 03 22:07:13.740176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.740145    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-29dbd9\"/\"openshift-service-ca.crt\""
Nov 03 22:07:13.740397 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.740173    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-29dbd9\"/\"default-dockercfg-bhh7c\""
Nov 03 22:07:13.740397 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.740235    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-29dbd9\"/\"auth-test-model-29dbd9-kserve-self-signed-certs\""
Nov 03 22:07:13.742960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.742935    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"]
Nov 03 22:07:13.815577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.815039    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.815577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.815110    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mkk\" (UniqueName: \"kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.815577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.815149    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.815577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.815193    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.815577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.815275    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.815577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.815385    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916418 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916276    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916418 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916383    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916671 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916481    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916671 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916524    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mkk\" (UniqueName: \"kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916671 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916556    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916671 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916601    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916734    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.916968 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916948    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.917028 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.916992    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.919046 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.919016    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.919208 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.919194    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:13.973641 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:13.973607    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mkk\" (UniqueName: \"kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk\") pod \"auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") " pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:14.045842 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:14.045776    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"]
Nov 03 22:07:14.049284 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:14.049259    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eea77b4-rs9pk"]
Nov 03 22:07:14.250508 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:14.250466    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:14.627607 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:14.627580    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"]
Nov 03 22:07:14.629437 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:07:14.629407    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod528e9a14_1d93_4f9b_adab_103d915d056c.slice/crio-04187e6e5cbd5da5992a4f783ed3469fb34ba7b4e9cadb10ab834676985cb165 WatchSource:0}: Error finding container 04187e6e5cbd5da5992a4f783ed3469fb34ba7b4e9cadb10ab834676985cb165: Status 404 returned error can't find the container with id 04187e6e5cbd5da5992a4f783ed3469fb34ba7b4e9cadb10ab834676985cb165
Nov 03 22:07:15.117574 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:15.117538    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" event={"ID":"528e9a14-1d93-4f9b-adab-103d915d056c","Type":"ContainerStarted","Data":"a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604"}
Nov 03 22:07:15.117988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:15.117587    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" event={"ID":"528e9a14-1d93-4f9b-adab-103d915d056c","Type":"ContainerStarted","Data":"04187e6e5cbd5da5992a4f783ed3469fb34ba7b4e9cadb10ab834676985cb165"}
Nov 03 22:07:15.146800 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:15.146760    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2b3dc4-2815-4a89-a9b8-606e37173768" path="/var/lib/kubelet/pods/5d2b3dc4-2815-4a89-a9b8-606e37173768/volumes"
Nov 03 22:07:15.232854 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:15.232801    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" podStartSLOduration=2.232786998 podStartE2EDuration="2.232786998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:15.230868869 +0000 UTC m=+3106.214865430" watchObservedRunningTime="2026-11-03 22:07:15.232786998 +0000 UTC m=+3106.216783598"
Nov 03 22:07:15.251225 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:15.251192    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:22.043930 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:22.043894    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"]
Nov 03 22:07:22.093095 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:22.093054    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d58dfae-x9ccs"]
Nov 03 22:07:22.436885 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:07:22.436842    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:07:23.137229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:23.137193    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a318f787-035e-4ce0-b268-40c0b357c776" path="/var/lib/kubelet/pods/a318f787-035e-4ce0-b268-40c0b357c776/volumes"
Nov 03 22:07:24.255497 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:24.255462    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:24.269285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:24.269252    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:25.190380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:25.190321    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:07:30.151231 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:30.151195    5244 scope.go:122] "RemoveContainer" containerID="04cc1fd44746fe720b7bf9944702788f8a5cad2e896f84cb47bdbf5d219f3d83"
Nov 03 22:07:30.161263 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:30.161237    5244 scope.go:122] "RemoveContainer" containerID="8ccf85ba0cf8608c4c6eaf1eb3d4a8bed8ca2f9d312895004252a3035e823480"
Nov 03 22:07:30.170476 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:30.170449    5244 scope.go:122] "RemoveContainer" containerID="88ee3a2d03f4eabd8a7fe952d04386564b754043b95c49d0f342b18e28b09d4d"
Nov 03 22:07:30.180179 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:30.180146    5244 scope.go:122] "RemoveContainer" containerID="23d9da974931e012c16d875f8a598d69c3284b7872da7e5d565112c359a958c9"
Nov 03 22:07:30.195603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:30.195570    5244 scope.go:122] "RemoveContainer" containerID="b747bcbfd2328720ada18948c59956f135d093cb2a879022bd87f534dadf7db7"
Nov 03 22:07:32.045609 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:32.045575    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"]
Nov 03 22:07:32.094923 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:32.094887    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4cc6926-58ksl"]
Nov 03 22:07:33.135921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:33.135887    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ecd850-8a21-4ba0-bbdf-d0bd18809ea4" path="/var/lib/kubelet/pods/36ecd850-8a21-4ba0-bbdf-d0bd18809ea4/volumes"
Nov 03 22:07:34.044859 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:34.044823    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"]
Nov 03 22:07:34.094307 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:34.094265    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1257e9a-hlkr7"]
Nov 03 22:07:35.136808 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:35.136777    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6318b399-f69d-4817-9980-b9ec438771de" path="/var/lib/kubelet/pods/6318b399-f69d-4817-9980-b9ec438771de/volumes"
Nov 03 22:07:40.269835 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:40.269802    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:07:51.045216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.045105    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"]
Nov 03 22:07:51.094064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.094030    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f028c9e1-ncd4d"]
Nov 03 22:07:51.136709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.136673    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdaa3d96-17f8-4296-a8e8-cfe87797ad52" path="/var/lib/kubelet/pods/cdaa3d96-17f8-4296-a8e8-cfe87797ad52/volumes"
Nov 03 22:07:51.370201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.370107    5244 generic.go:356] "Generic (PLEG): container finished" podID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerID="cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0" exitCode=0
Nov 03 22:07:51.370201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.370183    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" event={"ID":"17edd6f0-d922-46ea-8ad7-428466cfdab9","Type":"ContainerDied","Data":"cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"}
Nov 03 22:07:51.370653 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.370639    5244 scope.go:122] "RemoveContainer" containerID="cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"
Nov 03 22:07:51.372577 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.372545    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:07:51.372666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.372623    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:07:51.372666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:51.372655    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:07:52.378527 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:52.378496    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" event={"ID":"17edd6f0-d922-46ea-8ad7-428466cfdab9","Type":"ContainerStarted","Data":"79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b"}
Nov 03 22:07:52.463674 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:07:52.463635    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:07:53.021796 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:53.021763    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:07:53.022866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:53.022842    5244 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 22:07:53.022939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:53.022923    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 22:07:54.021839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:54.021807    5244 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 22:07:54.022271 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:54.021861    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 22:07:55.021972 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:55.021944    5244 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 22:07:55.022465 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:55.022440    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 22:07:55.659583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:55.659541    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:07:56.021427 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:56.021393    5244 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 22:07:56.021624 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:56.021446    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 22:07:57.021355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:57.021297    5244 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 22:07:57.021785 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:57.021374    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 22:07:58.098945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:58.098911    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:07:58.105938 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:07:58.105910    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:08:03.863317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:03.863276    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"]
Nov 03 22:08:03.863851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:03.863589    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" containerID="cri-o://3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1" gracePeriod=30
Nov 03 22:08:08.863633 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:08.863540    5244 patch_prober.go:29] interesting pod/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:08:08.863633 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:08.863613    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:08:09.117510 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.117447    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:08:09.157049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157020    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157061    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157106    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157173    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157217    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157309    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157394    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157406    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:09.157478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157425    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157691 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157478    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo\") pod \"2220fc07-2e8a-4427-bade-be4bdd90add3\" (UID: \"2220fc07-2e8a-4427-bade-be4bdd90add3\") "
Nov 03 22:08:09.157691 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157512    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:09.157691 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157662    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:09.157993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157890    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:09.157993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157943    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:09.158139 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.157964    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.158186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.158162    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.158225 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.158191    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.159548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.159527    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:09.159757 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.159726    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:08:09.159918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.159899    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "kube-api-access-8fv6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:09.159991 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.159972    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token" pod "2220fc07-2e8a-4427-bade-be4bdd90add3" (UID: "2220fc07-2e8a-4427-bade-be4bdd90add3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:09.259720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.259679    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fv6g\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-kube-api-access-8fv6g\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.259720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.259711    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.259720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.259725    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.259986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.259740    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2220fc07-2e8a-4427-bade-be4bdd90add3-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.259986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.259751    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.259986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.259764    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2220fc07-2e8a-4427-bade-be4bdd90add3-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:09.489489 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.489438    5244 generic.go:356] "Generic (PLEG): container finished" podID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerID="3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1" exitCode=0
Nov 03 22:08:09.489723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.489497    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"
Nov 03 22:08:09.489723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.489529    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" event={"ID":"2220fc07-2e8a-4427-bade-be4bdd90add3","Type":"ContainerDied","Data":"3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1"}
Nov 03 22:08:09.489723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.489598    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9" event={"ID":"2220fc07-2e8a-4427-bade-be4bdd90add3","Type":"ContainerDied","Data":"9c76dee66f09d65663f6ec1355463218b6950718a506038a2afa94889a7d3410"}
Nov 03 22:08:09.489723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.489629    5244 scope.go:122] "RemoveContainer" containerID="3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1"
Nov 03 22:08:09.500660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.500632    5244 scope.go:122] "RemoveContainer" containerID="3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1"
Nov 03 22:08:09.500962 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:08:09.500941    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1\": container with ID starting with 3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1 not found: ID does not exist" containerID="3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1"
Nov 03 22:08:09.501015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.500974    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1"} err="failed to get container status \"3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1\": rpc error: code = NotFound desc = could not find container \"3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1\": container with ID starting with 3322e838ab2d2ad9272aca076a897d5a4cc7d0bb5ed81f8acb93d393570e15b1 not found: ID does not exist"
Nov 03 22:08:09.651226 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.651193    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"]
Nov 03 22:08:09.653110 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:09.653086    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-vtrz9"]
Nov 03 22:08:11.137454 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:11.137416    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" path="/var/lib/kubelet/pods/2220fc07-2e8a-4427-bade-be4bdd90add3/volumes"
Nov 03 22:08:12.465523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:12.465486    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"]
Nov 03 22:08:12.508375 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:12.508311    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy" containerID="cri-o://ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf" gracePeriod=30
Nov 03 22:08:17.767873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.767847    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:08:17.874241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874143    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874187    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874221    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874249    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874312    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874389    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874434    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874472    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.874583 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874540    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874592    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs\") pod \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\" (UID: \"93233c9c-b516-4ca1-97e1-42dcabe8e4c0\") "
Nov 03 22:08:17.874889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874658    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.874889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874681    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.875033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.874971    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.875033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.875013    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:08:17.875282 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.875263    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.875398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.875286    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.875398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.875300    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.875398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.875314    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.876692 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.876674    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:17.876991 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.876970    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "kube-api-access-w666n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:08:17.877243 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.877226    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:08:17.877288 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.877244    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy" pod "93233c9c-b516-4ca1-97e1-42dcabe8e4c0" (UID: "93233c9c-b516-4ca1-97e1-42dcabe8e4c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:08:17.977106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.977055    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.977106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.977093    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.977106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.977105    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w666n\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-kube-api-access-w666n\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.977106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.977115    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:17.977476 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:17.977124    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93233c9c-b516-4ca1-97e1-42dcabe8e4c0-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:08:18.550785 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.550748    5244 generic.go:356] "Generic (PLEG): container finished" podID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerID="ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf" exitCode=0
Nov 03 22:08:18.550987 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.550804    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"
Nov 03 22:08:18.550987 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.550829    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" event={"ID":"93233c9c-b516-4ca1-97e1-42dcabe8e4c0","Type":"ContainerDied","Data":"ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf"}
Nov 03 22:08:18.550987 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.550882    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88" event={"ID":"93233c9c-b516-4ca1-97e1-42dcabe8e4c0","Type":"ContainerDied","Data":"9e8e705f793b29dafb4c5167adc77b41009170ed7ea9b7d5b904a79bfe375f95"}
Nov 03 22:08:18.550987 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.550903    5244 scope.go:122] "RemoveContainer" containerID="ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf"
Nov 03 22:08:18.561573 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.561556    5244 scope.go:122] "RemoveContainer" containerID="ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf"
Nov 03 22:08:18.561869 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:08:18.561851    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf\": container with ID starting with ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf not found: ID does not exist" containerID="ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf"
Nov 03 22:08:18.561923 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.561888    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf"} err="failed to get container status \"ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf\": rpc error: code = NotFound desc = could not find container \"ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf\": container with ID starting with ccff6d4359c5719124086a11fcc2e4affc2424cac584cb0c7e8ad360501727bf not found: ID does not exist"
Nov 03 22:08:18.713978 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.713943    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"]
Nov 03 22:08:18.763282 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:18.763246    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-tzx88"]
Nov 03 22:08:19.136297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:19.136263    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" path="/var/lib/kubelet/pods/93233c9c-b516-4ca1-97e1-42dcabe8e4c0/volumes"
Nov 03 22:08:22.436306 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:08:22.436267    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:08:30.239805 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.239770    5244 scope.go:122] "RemoveContainer" containerID="b9a037c56f1712f9635648250382db94322be208484dc7b661caf7bd6eea93e7"
Nov 03 22:08:30.249425 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.249401    5244 scope.go:122] "RemoveContainer" containerID="5fe79f9cb6a3ec9301bece13ed0f0f467da0af84df1c98afa4c2db143e8e7dea"
Nov 03 22:08:30.258816 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.258796    5244 scope.go:122] "RemoveContainer" containerID="1bc645ef2020fb755b4e3893a837857e393a463e83d9ae4d87ce5896656c5ed5"
Nov 03 22:08:30.268318 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.268287    5244 scope.go:122] "RemoveContainer" containerID="a2537c9e0f2bf660ca2d378c0edd6f86fcf9fd816ceb0f9d924391963cc368fc"
Nov 03 22:08:30.278113 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.278088    5244 scope.go:122] "RemoveContainer" containerID="cdfde7cba099c4e54415a68b7f9a107e5ba3ef5a2a4bf6529b1ba910453486c2"
Nov 03 22:08:30.287766 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.287744    5244 scope.go:122] "RemoveContainer" containerID="c3dd3b23e4f6a0c8f12a30498eddecd982b57963c35e29b4111d367dc2321e1a"
Nov 03 22:08:30.296312 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.296288    5244 scope.go:122] "RemoveContainer" containerID="bb750400dbe349ad54a952ad2fc38eceb4af7f800e228675d487fb608fab588a"
Nov 03 22:08:30.306587 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.306563    5244 scope.go:122] "RemoveContainer" containerID="7891e2b2a38fd70cfe81f2d8d7b30ddb0b8243de80974bd8e6b6cb91e593814d"
Nov 03 22:08:30.315541 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.315519    5244 scope.go:122] "RemoveContainer" containerID="d9e18f984d8b161ce7a49c2e48d23ccf7b26f0008c73f3d0aab0e5d7229a61ec"
Nov 03 22:08:30.324928 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:30.324904    5244 scope.go:122] "RemoveContainer" containerID="ebd1a969ef5804e51a51c71fc0de54b3fc9ce27efcd151b5742f3e6ed3e2d0a7"
Nov 03 22:08:33.357908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:08:33.357871    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:08:52.437185 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:08:52.437136    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:09:14.812223 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812188    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"]
Nov 03 22:09:14.812779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812706    5244 cpu_manager.go:414] "containerMap: removing container" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy"
Nov 03 22:09:14.812779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812720    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy"
Nov 03 22:09:14.812779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812744    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy"
Nov 03 22:09:14.812779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812749    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy"
Nov 03 22:09:14.812939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812821    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="93233c9c-b516-4ca1-97e1-42dcabe8e4c0" containerName="istio-proxy"
Nov 03 22:09:14.812939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.812830    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2220fc07-2e8a-4427-bade-be4bdd90add3" containerName="istio-proxy"
Nov 03 22:09:14.816096 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.816080    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:14.873362 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.873306    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"]
Nov 03 22:09:14.920120 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.920088    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:09:14.920120 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.920124    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:09:14.941600 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.941568    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:14.941736 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.941701    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:14.941777 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:14.941749    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrws\" (UniqueName: \"kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.042579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.042542    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.042755 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.042644    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.042755 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.042682    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrws\" (UniqueName: \"kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.043392 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.043373    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.045089 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.045070    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.098947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.098862    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrws\" (UniqueName: \"kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws\") pod \"maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.128826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.128781    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:15.503175 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.503151    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"]
Nov 03 22:09:15.505583 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:09:15.505541    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87810d5d_2f64_4729_bb35_efb6185899dc.slice/crio-97bd05eb2d56e71ae0ffa23d8be56d88782b9e95429f97a8d61c3afa62d24c0e WatchSource:0}: Error finding container 97bd05eb2d56e71ae0ffa23d8be56d88782b9e95429f97a8d61c3afa62d24c0e: Status 404 returned error can't find the container with id 97bd05eb2d56e71ae0ffa23d8be56d88782b9e95429f97a8d61c3afa62d24c0e
Nov 03 22:09:15.944845 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.944802    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7" event={"ID":"87810d5d-2f64-4729-bb35-efb6185899dc","Type":"ContainerStarted","Data":"d922bd9756d81cecbd00dcff59955a8ab924f8befdb59e084d29f21e6d537e5c"}
Nov 03 22:09:15.945430 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:15.944876    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7" event={"ID":"87810d5d-2f64-4729-bb35-efb6185899dc","Type":"ContainerStarted","Data":"97bd05eb2d56e71ae0ffa23d8be56d88782b9e95429f97a8d61c3afa62d24c0e"}
Nov 03 22:09:16.058696 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:16.058645    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7" podStartSLOduration=2.058631839 podStartE2EDuration="2.058631839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:16.056899573 +0000 UTC m=+3227.040896189" watchObservedRunningTime="2026-11-03 22:09:16.058631839 +0000 UTC m=+3227.042628398"
Nov 03 22:09:16.951726 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:16.951683    5244 generic.go:356] "Generic (PLEG): container finished" podID="87810d5d-2f64-4729-bb35-efb6185899dc" containerID="d922bd9756d81cecbd00dcff59955a8ab924f8befdb59e084d29f21e6d537e5c" exitCode=0
Nov 03 22:09:16.952128 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:16.951762    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7" event={"ID":"87810d5d-2f64-4729-bb35-efb6185899dc","Type":"ContainerDied","Data":"d922bd9756d81cecbd00dcff59955a8ab924f8befdb59e084d29f21e6d537e5c"}
Nov 03 22:09:18.099443 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.099418    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:18.177373 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.177339    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca\") pod \"87810d5d-2f64-4729-bb35-efb6185899dc\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") "
Nov 03 22:09:18.177576 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.177390    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws\" (UniqueName: \"kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws\") pod \"87810d5d-2f64-4729-bb35-efb6185899dc\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") "
Nov 03 22:09:18.177576 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.177437    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca\") pod \"87810d5d-2f64-4729-bb35-efb6185899dc\" (UID: \"87810d5d-2f64-4729-bb35-efb6185899dc\") "
Nov 03 22:09:18.177811 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.177784    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca" pod "87810d5d-2f64-4729-bb35-efb6185899dc" (UID: "87810d5d-2f64-4729-bb35-efb6185899dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:18.178020 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.178000    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:18.179824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.179793    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca" pod "87810d5d-2f64-4729-bb35-efb6185899dc" (UID: "87810d5d-2f64-4729-bb35-efb6185899dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:09:18.179824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.179818    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws" pod "87810d5d-2f64-4729-bb35-efb6185899dc" (UID: "87810d5d-2f64-4729-bb35-efb6185899dc"). InnerVolumeSpecName "kube-api-access-cqrws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:18.279483 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.279396    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87810d5d-2f64-4729-bb35-efb6185899dc-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:18.279483 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.279427    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqrws\" (UniqueName: \"kubernetes.io/projected/87810d5d-2f64-4729-bb35-efb6185899dc-kube-api-access-cqrws\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:18.966316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.966283    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"
Nov 03 22:09:18.966517 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.966282    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7" event={"ID":"87810d5d-2f64-4729-bb35-efb6185899dc","Type":"ContainerDied","Data":"97bd05eb2d56e71ae0ffa23d8be56d88782b9e95429f97a8d61c3afa62d24c0e"}
Nov 03 22:09:18.966517 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:18.966443    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97bd05eb2d56e71ae0ffa23d8be56d88782b9e95429f97a8d61c3afa62d24c0e"
Nov 03 22:09:20.054259 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:20.054222    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:09:22.438027 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:09:22.437985    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:09:22.694147 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.694063    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"]
Nov 03 22:09:22.694539 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.694522    5244 cpu_manager.go:414] "containerMap: removing container" podUID="87810d5d-2f64-4729-bb35-efb6185899dc" containerName="revoke-keys"
Nov 03 22:09:22.694539 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.694539    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87810d5d-2f64-4729-bb35-efb6185899dc" containerName="revoke-keys"
Nov 03 22:09:22.694656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.694647    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="87810d5d-2f64-4729-bb35-efb6185899dc" containerName="revoke-keys"
Nov 03 22:09:22.698004 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.697987    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.801750 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.801717    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-0d45ba\"/\"kube-root-ca.crt\""
Nov 03 22:09:22.801750 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.801744    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-0d45ba\"/\"openshift-service-ca.crt\""
Nov 03 22:09:22.801967 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.801779    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-0d45ba\"/\"auth-test-model-0d45ba-kserve-self-signed-certs\""
Nov 03 22:09:22.802193 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.802170    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-0d45ba\"/\"default-dockercfg-kxnbc\""
Nov 03 22:09:22.804195 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.804173    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"]
Nov 03 22:09:22.833520 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.833483    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.833701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.833527    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.833701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.833654    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.833792 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.833742    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.833792 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.833777    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.833918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.833899    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlpl\" (UniqueName: \"kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935167 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935132    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935167 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935176    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935201    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935246    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935269    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935361    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlpl\" (UniqueName: \"kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935710 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935669    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935858    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.935950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.935919    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.937831 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.937806    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.937998 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.937982    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:22.992745 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:22.992661    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlpl\" (UniqueName: \"kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl\") pod \"auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") " pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:23.009910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:23.009867    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:23.396318 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:23.396288    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"]
Nov 03 22:09:23.398963 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:09:23.398932    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ce8a87_1937_4c86_959f_1edf618feefe.slice/crio-760027d6fdaae090c96983c550b50a93780778b532ebcef02c500c2150566417 WatchSource:0}: Error finding container 760027d6fdaae090c96983c550b50a93780778b532ebcef02c500c2150566417: Status 404 returned error can't find the container with id 760027d6fdaae090c96983c550b50a93780778b532ebcef02c500c2150566417
Nov 03 22:09:24.004099 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:24.004062    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" event={"ID":"64ce8a87-1937-4c86-959f-1edf618feefe","Type":"ContainerStarted","Data":"f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53"}
Nov 03 22:09:24.004552 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:24.004113    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" event={"ID":"64ce8a87-1937-4c86-959f-1edf618feefe","Type":"ContainerStarted","Data":"760027d6fdaae090c96983c550b50a93780778b532ebcef02c500c2150566417"}
Nov 03 22:09:24.010582 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:24.010546    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:24.116040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:24.115977    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" podStartSLOduration=2.11596419 podStartE2EDuration="2.11596419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:24.115913901 +0000 UTC m=+3235.099910490" watchObservedRunningTime="2026-11-03 22:09:24.11596419 +0000 UTC m=+3235.099960730"
Nov 03 22:09:27.378139 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:27.378100    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"]
Nov 03 22:09:30.396649 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:30.396616    5244 scope.go:122] "RemoveContainer" containerID="661f1755efdf197eac2e70ac1f37a1fe5c522b9440fc6c550ff2e8eb562cdc0e"
Nov 03 22:09:30.406289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:30.406262    5244 scope.go:122] "RemoveContainer" containerID="4ad3152d895924c798dba2487306fa3327aac860522356362a5e82d15b8d5dac"
Nov 03 22:09:30.418474 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:30.418454    5244 scope.go:122] "RemoveContainer" containerID="c0db690058673fab82c88bc4b5d7c43cf9b127da28b95744769505e501b5106e"
Nov 03 22:09:32.378948 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.378908    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8" podUID="615e25f0-8693-4288-ad4f-b3b3c1b43608" containerName="payload-processing" containerID="cri-o://13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77" gracePeriod=25
Nov 03 22:09:32.643121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.643054    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:09:32.741429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.741389    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872\" (UniqueName: \"kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872\") pod \"615e25f0-8693-4288-ad4f-b3b3c1b43608\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") "
Nov 03 22:09:32.741594 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.741541    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume\") pod \"615e25f0-8693-4288-ad4f-b3b3c1b43608\" (UID: \"615e25f0-8693-4288-ad4f-b3b3c1b43608\") "
Nov 03 22:09:32.741928 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.741901    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume" pod "615e25f0-8693-4288-ad4f-b3b3c1b43608" (UID: "615e25f0-8693-4288-ad4f-b3b3c1b43608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:09:32.743614 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.743592    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872" pod "615e25f0-8693-4288-ad4f-b3b3c1b43608" (UID: "615e25f0-8693-4288-ad4f-b3b3c1b43608"). InnerVolumeSpecName "kube-api-access-d8872". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:09:32.843446 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.843405    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/615e25f0-8693-4288-ad4f-b3b3c1b43608-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:32.843446 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:32.843440    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8872\" (UniqueName: \"kubernetes.io/projected/615e25f0-8693-4288-ad4f-b3b3c1b43608-kube-api-access-d8872\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:09:33.010512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.010478    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:33.024006 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.023973    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:33.056153 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.056117    5244 generic.go:356] "Generic (PLEG): container finished" podID="615e25f0-8693-4288-ad4f-b3b3c1b43608" containerID="13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77" exitCode=0
Nov 03 22:09:33.056357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.056171    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"
Nov 03 22:09:33.056357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.056206    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8" event={"ID":"615e25f0-8693-4288-ad4f-b3b3c1b43608","Type":"ContainerDied","Data":"13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77"}
Nov 03 22:09:33.056357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.056261    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8" event={"ID":"615e25f0-8693-4288-ad4f-b3b3c1b43608","Type":"ContainerDied","Data":"d54729df8213471e906143f566d490598c588a1bca527ea401e45f05e9f4f74e"}
Nov 03 22:09:33.056357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.056288    5244 scope.go:122] "RemoveContainer" containerID="13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77"
Nov 03 22:09:33.067064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.067037    5244 scope.go:122] "RemoveContainer" containerID="13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77"
Nov 03 22:09:33.067472 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:09:33.067418    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77\": container with ID starting with 13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77 not found: ID does not exist" containerID="13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77"
Nov 03 22:09:33.067578 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.067484    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77"} err="failed to get container status \"13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77\": rpc error: code = NotFound desc = could not find container \"13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77\": container with ID starting with 13d0b047a55db6af107636439a970ab830b5f3ceb90916bd42d3bdc6c3d99d77 not found: ID does not exist"
Nov 03 22:09:33.068903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.068886    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:09:33.364960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.364884    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"]
Nov 03 22:09:33.368063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:33.368031    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-27a3c3-786dc4844d-95tl8"]
Nov 03 22:09:35.143523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:35.143486    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615e25f0-8693-4288-ad4f-b3b3c1b43608" path="/var/lib/kubelet/pods/615e25f0-8693-4288-ad4f-b3b3c1b43608/volumes"
Nov 03 22:09:40.116012 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.115919    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"]
Nov 03 22:09:40.116588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.116566    5244 cpu_manager.go:414] "containerMap: removing container" podUID="615e25f0-8693-4288-ad4f-b3b3c1b43608" containerName="payload-processing"
Nov 03 22:09:40.116673 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.116590    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615e25f0-8693-4288-ad4f-b3b3c1b43608" containerName="payload-processing"
Nov 03 22:09:40.116756 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.116743    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="615e25f0-8693-4288-ad4f-b3b3c1b43608" containerName="payload-processing"
Nov 03 22:09:40.121142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.121122    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.129700 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.129678    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.129792 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.129734    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.129912 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.129900    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.129953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.129922    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssbw\" (UniqueName: \"kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.129953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.129940    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.130029 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.129966    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.178069 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.178038    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"]
Nov 03 22:09:40.199507 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.199473    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e15bbc\"/\"default-dockercfg-lt5tt\""
Nov 03 22:09:40.224091 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.224048    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e15bbc\"/\"openshift-service-ca.crt\""
Nov 03 22:09:40.224285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.224163    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e15bbc\"/\"auth-test-model-e15bbc-kserve-self-signed-certs\""
Nov 03 22:09:40.224285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.224185    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e15bbc\"/\"kube-root-ca.crt\""
Nov 03 22:09:40.240051 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240007    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240063    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssbw\" (UniqueName: \"kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240092    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240136    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240194    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240463 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240440    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240705 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240682    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240881 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240792    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.240972 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.240882    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.242890 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.242868    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.242971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.242940    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.325145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.325105    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssbw\" (UniqueName: \"kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw\") pod \"auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") " pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.432686 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.432661    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:40.818260 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:40.818149    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"]
Nov 03 22:09:40.832309 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:09:40.832271    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c781c17_58fd_41a3_9cc5_c779c23c4f65.slice/crio-b4842da76b8e721a74e8af506878fe64f83a39bc58a4a4d879c8b41dbebd63ec WatchSource:0}: Error finding container b4842da76b8e721a74e8af506878fe64f83a39bc58a4a4d879c8b41dbebd63ec: Status 404 returned error can't find the container with id b4842da76b8e721a74e8af506878fe64f83a39bc58a4a4d879c8b41dbebd63ec
Nov 03 22:09:41.106041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:41.105941    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" event={"ID":"4c781c17-58fd-41a3-9cc5-c779c23c4f65","Type":"ContainerStarted","Data":"ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae"}
Nov 03 22:09:41.106041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:41.105994    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" event={"ID":"4c781c17-58fd-41a3-9cc5-c779c23c4f65","Type":"ContainerStarted","Data":"b4842da76b8e721a74e8af506878fe64f83a39bc58a4a4d879c8b41dbebd63ec"}
Nov 03 22:09:41.219588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:41.219534    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" podStartSLOduration=1.219518506 podStartE2EDuration="1.219518506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:09:41.218030586 +0000 UTC m=+3252.202027171" watchObservedRunningTime="2026-11-03 22:09:41.219518506 +0000 UTC m=+3252.203515065"
Nov 03 22:09:41.433650 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:41.433612    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:50.432842 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:50.432814    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:50.446922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:50.446898    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:51.173165 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:51.173087    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:09:52.436903 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:09:52.436859    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:09:59.756774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:09:59.756743    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:10:08.778804 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.778770    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"]
Nov 03 22:10:08.782687 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.782669    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.838971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.838938    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"]
Nov 03 22:10:08.867412 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.867371    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqrl\" (UniqueName: \"kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.867588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.867557    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.867630 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.867592    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.885912 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.885867    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:10:08.885912 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.885904    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:10:08.968903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.968860    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqrl\" (UniqueName: \"kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.969112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.968988    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.969112 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.969009    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.969727 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.969701    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:08.971581 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:08.971562    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:09.027665 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:09.027629    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqrl\" (UniqueName: \"kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl\") pod \"maas-api-revoke-sub-6175ce20-2tb9t\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:09.096567 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:09.096488    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:09.472164 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:09.472136    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"]
Nov 03 22:10:09.475307 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:10:09.475278    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ec4513_7ac4_4147_9a77_189d818b063d.slice/crio-0bf2cbfe9994bbe203e921b3ea0138bf41f644cb5b33e38184856e14a2a0ac01 WatchSource:0}: Error finding container 0bf2cbfe9994bbe203e921b3ea0138bf41f644cb5b33e38184856e14a2a0ac01: Status 404 returned error can't find the container with id 0bf2cbfe9994bbe203e921b3ea0138bf41f644cb5b33e38184856e14a2a0ac01
Nov 03 22:10:10.268899 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:10.268864    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t" event={"ID":"f6ec4513-7ac4-4147-9a77-189d818b063d","Type":"ContainerStarted","Data":"7b482bae56e093815c9075846691054031099f9bdc1897a3706df8d0436323f1"}
Nov 03 22:10:10.269351 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:10.268918    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t" event={"ID":"f6ec4513-7ac4-4147-9a77-189d818b063d","Type":"ContainerStarted","Data":"0bf2cbfe9994bbe203e921b3ea0138bf41f644cb5b33e38184856e14a2a0ac01"}
Nov 03 22:10:10.382111 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:10.382061    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t" podStartSLOduration=2.382044545 podStartE2EDuration="2.382044545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:10:10.380669124 +0000 UTC m=+3281.364665704" watchObservedRunningTime="2026-11-03 22:10:10.382044545 +0000 UTC m=+3281.366041103"
Nov 03 22:10:11.276838 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:11.276802    5244 generic.go:356] "Generic (PLEG): container finished" podID="f6ec4513-7ac4-4147-9a77-189d818b063d" containerID="7b482bae56e093815c9075846691054031099f9bdc1897a3706df8d0436323f1" exitCode=0
Nov 03 22:10:11.277347 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:11.276885    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t" event={"ID":"f6ec4513-7ac4-4147-9a77-189d818b063d","Type":"ContainerDied","Data":"7b482bae56e093815c9075846691054031099f9bdc1897a3706df8d0436323f1"}
Nov 03 22:10:12.429545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.429519    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:12.611791 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.611708    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca\") pod \"f6ec4513-7ac4-4147-9a77-189d818b063d\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") "
Nov 03 22:10:12.611957 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.611942    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca\") pod \"f6ec4513-7ac4-4147-9a77-189d818b063d\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") "
Nov 03 22:10:12.612006 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.611973    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl\" (UniqueName: \"kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl\") pod \"f6ec4513-7ac4-4147-9a77-189d818b063d\" (UID: \"f6ec4513-7ac4-4147-9a77-189d818b063d\") "
Nov 03 22:10:12.612344 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.612301    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca" pod "f6ec4513-7ac4-4147-9a77-189d818b063d" (UID: "f6ec4513-7ac4-4147-9a77-189d818b063d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:12.614073 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.614048    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca" pod "f6ec4513-7ac4-4147-9a77-189d818b063d" (UID: "f6ec4513-7ac4-4147-9a77-189d818b063d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:10:12.614352 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.614312    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl" pod "f6ec4513-7ac4-4147-9a77-189d818b063d" (UID: "f6ec4513-7ac4-4147-9a77-189d818b063d"). InnerVolumeSpecName "kube-api-access-mlqrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:12.713294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.713251    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:12.713294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.713283    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlqrl\" (UniqueName: \"kubernetes.io/projected/f6ec4513-7ac4-4147-9a77-189d818b063d-kube-api-access-mlqrl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:12.713294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:12.713295    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6ec4513-7ac4-4147-9a77-189d818b063d-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:13.289887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:13.289850    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"
Nov 03 22:10:13.289887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:13.289877    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t" event={"ID":"f6ec4513-7ac4-4147-9a77-189d818b063d","Type":"ContainerDied","Data":"0bf2cbfe9994bbe203e921b3ea0138bf41f644cb5b33e38184856e14a2a0ac01"}
Nov 03 22:10:13.290098 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:13.289927    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf2cbfe9994bbe203e921b3ea0138bf41f644cb5b33e38184856e14a2a0ac01"
Nov 03 22:10:18.793667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:18.793631    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"]
Nov 03 22:10:18.802902 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:18.802874    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-27a3c3-8jzr7"]
Nov 03 22:10:19.135844 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.135763    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87810d5d-2f64-4729-bb35-efb6185899dc" path="/var/lib/kubelet/pods/87810d5d-2f64-4729-bb35-efb6185899dc/volumes"
Nov 03 22:10:19.599046 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.599013    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"]
Nov 03 22:10:19.599516 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.599503    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f6ec4513-7ac4-4147-9a77-189d818b063d" containerName="revoke-keys"
Nov 03 22:10:19.599632 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.599519    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ec4513-7ac4-4147-9a77-189d818b063d" containerName="revoke-keys"
Nov 03 22:10:19.599632 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.599607    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ec4513-7ac4-4147-9a77-189d818b063d" containerName="revoke-keys"
Nov 03 22:10:19.603060 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.603043    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.660759 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.660710    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"]
Nov 03 22:10:19.698957 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.698921    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.699127 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.698983    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdt8\" (UniqueName: \"kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.699127 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.699063    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.707417 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.707392    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:10:19.707529 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.707441    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:10:19.791081 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.791038    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"]
Nov 03 22:10:19.791314 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.791282    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy" containerID="cri-o://084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc" gracePeriod=30
Nov 03 22:10:19.800613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.800585    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.801029 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.800653    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdt8\" (UniqueName: \"kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.801029 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.800682    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.801590 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.801550    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.803322 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.803300    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.857515 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.857442    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdt8\" (UniqueName: \"kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8\") pod \"maas-api-revoke-sub-e33e35cc-c88hp\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:19.916555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:19.916528    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:20.293248 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:20.293148    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"]
Nov 03 22:10:20.295824 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:10:20.295798    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257d40d9_7d49_4ab0_bf3c_15626215e08a.slice/crio-d2f69a568d03358ded5e4b0f7a226d3fb082620e1be3706058c04cb4376cff52 WatchSource:0}: Error finding container d2f69a568d03358ded5e4b0f7a226d3fb082620e1be3706058c04cb4376cff52: Status 404 returned error can't find the container with id d2f69a568d03358ded5e4b0f7a226d3fb082620e1be3706058c04cb4376cff52
Nov 03 22:10:20.331393 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:20.331362    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp" event={"ID":"257d40d9-7d49-4ab0-bf3c-15626215e08a","Type":"ContainerStarted","Data":"d2f69a568d03358ded5e4b0f7a226d3fb082620e1be3706058c04cb4376cff52"}
Nov 03 22:10:21.346678 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:21.346639    5244 generic.go:356] "Generic (PLEG): container finished" podID="257d40d9-7d49-4ab0-bf3c-15626215e08a" containerID="6ea5dd55a23b4de4401b673b28f4cb7aa8ba413bc1ae8f5ed82e92eab9db2916" exitCode=0
Nov 03 22:10:21.347289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:21.346725    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp" event={"ID":"257d40d9-7d49-4ab0-bf3c-15626215e08a","Type":"ContainerDied","Data":"6ea5dd55a23b4de4401b673b28f4cb7aa8ba413bc1ae8f5ed82e92eab9db2916"}
Nov 03 22:10:22.448110 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:10:22.448063    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:10:22.502989 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.502963    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:22.635865 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.635771    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca\") pod \"257d40d9-7d49-4ab0-bf3c-15626215e08a\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") "
Nov 03 22:10:22.635865 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.635828    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8\" (UniqueName: \"kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8\") pod \"257d40d9-7d49-4ab0-bf3c-15626215e08a\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") "
Nov 03 22:10:22.636089 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.635992    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca\") pod \"257d40d9-7d49-4ab0-bf3c-15626215e08a\" (UID: \"257d40d9-7d49-4ab0-bf3c-15626215e08a\") "
Nov 03 22:10:22.636420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.636392    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca" pod "257d40d9-7d49-4ab0-bf3c-15626215e08a" (UID: "257d40d9-7d49-4ab0-bf3c-15626215e08a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:22.638229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.638196    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8" pod "257d40d9-7d49-4ab0-bf3c-15626215e08a" (UID: "257d40d9-7d49-4ab0-bf3c-15626215e08a"). InnerVolumeSpecName "kube-api-access-fpdt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:22.638376 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.638267    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca" pod "257d40d9-7d49-4ab0-bf3c-15626215e08a" (UID: "257d40d9-7d49-4ab0-bf3c-15626215e08a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:10:22.737967 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.737929    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:22.737967 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.737959    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpdt8\" (UniqueName: \"kubernetes.io/projected/257d40d9-7d49-4ab0-bf3c-15626215e08a-kube-api-access-fpdt8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:22.737967 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:22.737970    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257d40d9-7d49-4ab0-bf3c-15626215e08a-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:23.359104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:23.359072    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"
Nov 03 22:10:23.359104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:23.359092    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp" event={"ID":"257d40d9-7d49-4ab0-bf3c-15626215e08a","Type":"ContainerDied","Data":"d2f69a568d03358ded5e4b0f7a226d3fb082620e1be3706058c04cb4376cff52"}
Nov 03 22:10:23.359382 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:23.359144    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f69a568d03358ded5e4b0f7a226d3fb082620e1be3706058c04cb4376cff52"
Nov 03 22:10:25.095823 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.095800    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:10:25.170092 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170059    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170137    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170250    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170422 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170323    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170488 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170434    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170488 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170457    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:25.170488 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170475    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170524    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170595    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170650    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b\") pod \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\" (UID: \"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0\") "
Nov 03 22:10:25.170774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170696    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:25.170774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.170742    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:10:25.171215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.171013    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:25.171215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.171055    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:25.171215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.171087    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.171215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.171104    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.171215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.171119    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.171215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.171133    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.172731 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.172705    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:10:25.172978 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.172954    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:25.173040 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.173027    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:10:25.173240 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.173226    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b" pod "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" (UID: "f3996a74-1c8b-475d-b042-5cb8b8c8c9a0"). InnerVolumeSpecName "kube-api-access-jhl6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:10:25.271950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.271912    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.271950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.271947    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.272156 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.271965    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhl6b\" (UniqueName: \"kubernetes.io/projected/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-kube-api-access-jhl6b\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.272156 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.271975    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.272156 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.271984    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:10:25.373223 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.373118    5244 generic.go:356] "Generic (PLEG): container finished" podID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerID="084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc" exitCode=0
Nov 03 22:10:25.373428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.373216    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"
Nov 03 22:10:25.373428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.373299    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" event={"ID":"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0","Type":"ContainerDied","Data":"084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc"}
Nov 03 22:10:25.373428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.373395    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b" event={"ID":"f3996a74-1c8b-475d-b042-5cb8b8c8c9a0","Type":"ContainerDied","Data":"77c47bad53251f06fbb19e9cfbe1c867b83e2fe4d108cb42af17d4162ac4ac95"}
Nov 03 22:10:25.373428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.373427    5244 scope.go:122] "RemoveContainer" containerID="084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc"
Nov 03 22:10:25.413803 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.413779    5244 scope.go:122] "RemoveContainer" containerID="084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc"
Nov 03 22:10:25.416387 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:10:25.415696    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc\": container with ID starting with 084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc not found: ID does not exist" containerID="084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc"
Nov 03 22:10:25.416387 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.415750    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc"} err="failed to get container status \"084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc\": rpc error: code = NotFound desc = could not find container \"084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc\": container with ID starting with 084ef2f196e590e127d2ca5de57c698c53fdd69f27ca82f9e835fa52ebedeadc not found: ID does not exist"
Nov 03 22:10:25.554797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.554715    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"]
Nov 03 22:10:25.555219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:25.555140    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2bs4b"]
Nov 03 22:10:27.135981 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:27.135931    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" path="/var/lib/kubelet/pods/f3996a74-1c8b-475d-b042-5cb8b8c8c9a0/volumes"
Nov 03 22:10:52.438243 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:10:52.438195    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:10:58.320297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:10:58.320257    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:11:22.436459 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:11:22.436377    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:11:44.051852 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:44.051818    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"]
Nov 03 22:11:44.052412 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:44.052024    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" containerID="cri-o://47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a" gracePeriod=30
Nov 03 22:11:48.921013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:48.920978    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:11:48.921481 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:48.921045    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:11:49.310540 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.310513    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:11:49.474990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.474897    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.474994    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475051    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475080    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475121    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475156    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475204 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475198    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475249    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475317    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo\") pod \"b68c8bfa-aca0-4316-843e-780c5e898259\" (UID: \"b68c8bfa-aca0-4316-843e-780c5e898259\") "
Nov 03 22:11:49.475589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475561    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:49.475754 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475667    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:49.475902 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475882    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.475966 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475908    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.475966 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475927    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:49.475966 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475956    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:11:49.476099 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.475983    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:49.478464 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.478426    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:49.478719 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.478695    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:11:49.479227 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.479201    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "kube-api-access-994nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:11:49.479355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.479263    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo" pod "b68c8bfa-aca0-4316-843e-780c5e898259" (UID: "b68c8bfa-aca0-4316-843e-780c5e898259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:11:49.577648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577610    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.577648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577640    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.577648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577649    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68c8bfa-aca0-4316-843e-780c5e898259-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.577648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577658    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.577934 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577667    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68c8bfa-aca0-4316-843e-780c5e898259-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.577934 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577677    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68c8bfa-aca0-4316-843e-780c5e898259-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.577934 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.577685    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-994nl\" (UniqueName: \"kubernetes.io/projected/b68c8bfa-aca0-4316-843e-780c5e898259-kube-api-access-994nl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:11:49.898768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.898674    5244 generic.go:356] "Generic (PLEG): container finished" podID="b68c8bfa-aca0-4316-843e-780c5e898259" containerID="47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a" exitCode=0
Nov 03 22:11:49.898768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.898721    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"
Nov 03 22:11:49.898992 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.898758    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" event={"ID":"b68c8bfa-aca0-4316-843e-780c5e898259","Type":"ContainerDied","Data":"47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a"}
Nov 03 22:11:49.898992 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.898829    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj" event={"ID":"b68c8bfa-aca0-4316-843e-780c5e898259","Type":"ContainerDied","Data":"947c2e41625cb796e90bdc4500873f7078705d1a80af7ed708d6f5a4b188036b"}
Nov 03 22:11:49.898992 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.898860    5244 scope.go:122] "RemoveContainer" containerID="47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a"
Nov 03 22:11:49.912509 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.910834    5244 scope.go:122] "RemoveContainer" containerID="47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a"
Nov 03 22:11:49.912509 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:11:49.911141    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a\": container with ID starting with 47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a not found: ID does not exist" containerID="47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a"
Nov 03 22:11:49.912509 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:49.911166    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a"} err="failed to get container status \"47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a\": rpc error: code = NotFound desc = could not find container \"47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a\": container with ID starting with 47a5f29a8a4fe08a1db40308ba1ca992cbab932a98ca1d3d44f3e2038f7c711a not found: ID does not exist"
Nov 03 22:11:50.046149 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:50.046112    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:11:50.062379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:50.062340    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"]
Nov 03 22:11:50.065017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:50.064991    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-5ldhj"]
Nov 03 22:11:51.136166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:11:51.136084    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" path="/var/lib/kubelet/pods/b68c8bfa-aca0-4316-843e-780c5e898259/volumes"
Nov 03 22:11:52.437242 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:11:52.437203    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:12:22.435868 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:12:22.435813    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:12:24.768093 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768059    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7673-0ba5b0"]
Nov 03 22:12:24.768561 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768517    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy"
Nov 03 22:12:24.768561 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768531    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy"
Nov 03 22:12:24.768561 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768550    5244 cpu_manager.go:414] "containerMap: removing container" podUID="257d40d9-7d49-4ab0-bf3c-15626215e08a" containerName="revoke-keys"
Nov 03 22:12:24.768561 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768556    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257d40d9-7d49-4ab0-bf3c-15626215e08a" containerName="revoke-keys"
Nov 03 22:12:24.768703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768579    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy"
Nov 03 22:12:24.768703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768584    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy"
Nov 03 22:12:24.768703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768648    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68c8bfa-aca0-4316-843e-780c5e898259" containerName="istio-proxy"
Nov 03 22:12:24.768703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768660    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3996a74-1c8b-475d-b042-5cb8b8c8c9a0" containerName="istio-proxy"
Nov 03 22:12:24.768703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.768667    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="257d40d9-7d49-4ab0-bf3c-15626215e08a" containerName="revoke-keys"
Nov 03 22:12:24.771894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.771877    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:24.825826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.825791    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7673-0ba5b0"]
Nov 03 22:12:24.872052 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.872012    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79hc\" (UniqueName: \"kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc\") pod \"test-curl-7673-0ba5b0\" (UID: \"58834766-efc4-46f1-873f-ae29ce0bf06c\") " pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:24.872595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.872578    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8dgqp\""
Nov 03 22:12:24.973417 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:24.973377    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79hc\" (UniqueName: \"kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc\") pod \"test-curl-7673-0ba5b0\" (UID: \"58834766-efc4-46f1-873f-ae29ce0bf06c\") " pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:25.029838 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:25.029765    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79hc\" (UniqueName: \"kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc\") pod \"test-curl-7673-0ba5b0\" (UID: \"58834766-efc4-46f1-873f-ae29ce0bf06c\") " pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:25.084251 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:25.084213    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:25.455782 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:25.455729    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7673-0ba5b0"]
Nov 03 22:12:25.458399 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:12:25.458363    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58834766_efc4_46f1_873f_ae29ce0bf06c.slice/crio-20974ade52ebe8e55ca53481444967da508b6b6bd3514a94f823f0d05d08adbc WatchSource:0}: Error finding container 20974ade52ebe8e55ca53481444967da508b6b6bd3514a94f823f0d05d08adbc: Status 404 returned error can't find the container with id 20974ade52ebe8e55ca53481444967da508b6b6bd3514a94f823f0d05d08adbc
Nov 03 22:12:26.101953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:26.101914    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7673-0ba5b0" event={"ID":"58834766-efc4-46f1-873f-ae29ce0bf06c","Type":"ContainerStarted","Data":"c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101"}
Nov 03 22:12:26.101953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:26.101966    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7673-0ba5b0" event={"ID":"58834766-efc4-46f1-873f-ae29ce0bf06c","Type":"ContainerStarted","Data":"20974ade52ebe8e55ca53481444967da508b6b6bd3514a94f823f0d05d08adbc"}
Nov 03 22:12:26.215808 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:26.215734    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7673-0ba5b0" podStartSLOduration=2.215717009 podStartE2EDuration="2.215717009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:26.214231966 +0000 UTC m=+3417.198228525" watchObservedRunningTime="2026-11-03 22:12:26.215717009 +0000 UTC m=+3417.199713568"
Nov 03 22:12:27.647503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:27.647465    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7673-0ba5b0"]
Nov 03 22:12:27.652294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:27.652266    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7673-0ba5b0"]
Nov 03 22:12:28.113392 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:28.113347    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7673-0ba5b0" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" containerName="test-curl-7673-0ba5b0" containerID="cri-o://c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101" gracePeriod=2
Nov 03 22:12:28.164245 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:12:28.164206    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7673-0ba5b0\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:29.193628 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:12:29.193581    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7673-0ba5b0\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:30.356935 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:30.356892    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:30.448458 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:30.441915    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc\" (UniqueName: \"kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc\") pod \"58834766-efc4-46f1-873f-ae29ce0bf06c\" (UID: \"58834766-efc4-46f1-873f-ae29ce0bf06c\") "
Nov 03 22:12:30.449151 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:30.449088    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc" pod "58834766-efc4-46f1-873f-ae29ce0bf06c" (UID: "58834766-efc4-46f1-873f-ae29ce0bf06c"). InnerVolumeSpecName "kube-api-access-l79hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:12:30.544856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:30.544815    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79hc\" (UniqueName: \"kubernetes.io/projected/58834766-efc4-46f1-873f-ae29ce0bf06c-kube-api-access-l79hc\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:12:31.136652 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:31.136612    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" path="/var/lib/kubelet/pods/58834766-efc4-46f1-873f-ae29ce0bf06c/volumes"
Nov 03 22:12:31.145664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:31.145634    5244 generic.go:356] "Generic (PLEG): container finished" podID="58834766-efc4-46f1-873f-ae29ce0bf06c" containerID="c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101" exitCode=137
Nov 03 22:12:31.145797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:31.145681    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7673-0ba5b0"
Nov 03 22:12:31.145797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:31.145732    5244 scope.go:122] "RemoveContainer" containerID="c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101"
Nov 03 22:12:31.154537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:31.154519    5244 scope.go:122] "RemoveContainer" containerID="c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101"
Nov 03 22:12:31.154828 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:12:31.154809    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101\": container with ID starting with c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101 not found: ID does not exist" containerID="c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101"
Nov 03 22:12:31.154876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:31.154839    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101"} err="failed to get container status \"c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101\": rpc error: code = NotFound desc = could not find container \"c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101\": container with ID starting with c3db62051c8143a9bfb5f47361c3d3e58bb091008d3e8960cc104d07b69d2101 not found: ID does not exist"
Nov 03 22:12:44.604791 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:12:44.604712    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:12:52.437153 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:12:52.437104    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:13:22.440876 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:13:22.440833    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:13:26.318957 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.318917    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"]
Nov 03 22:13:26.319480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.319393    5244 cpu_manager.go:414] "containerMap: removing container" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" containerName="test-curl-7673-0ba5b0"
Nov 03 22:13:26.319480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.319405    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" containerName="test-curl-7673-0ba5b0"
Nov 03 22:13:26.319480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.319476    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="58834766-efc4-46f1-873f-ae29ce0bf06c" containerName="test-curl-7673-0ba5b0"
Nov 03 22:13:26.322943 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.322925    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.428291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.428256    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:13:26.428494 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.428300    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:13:26.430587 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.430566    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"]
Nov 03 22:13:26.474062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.474027    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87zd\" (UniqueName: \"kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.474062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.474063    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.474282 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.474264    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.575739 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.575649    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.575739 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.575701    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87zd\" (UniqueName: \"kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.575994 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.575822    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.576462 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.576444    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.578241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.578218    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.631919 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.631886    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87zd\" (UniqueName: \"kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd\") pod \"maas-api-revoke-sub-da7f7d4b-l6pvf\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:26.635557 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:26.635536    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:27.012302 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:27.012276    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"]
Nov 03 22:13:27.015766 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:13:27.015738    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d65b028_da2f_4335_a853_ce90ae9b93a1.slice/crio-60a162e26a0a806cf66d6ecdd527bcc077c677e4209cfb125b7ad48203e56b4f WatchSource:0}: Error finding container 60a162e26a0a806cf66d6ecdd527bcc077c677e4209cfb125b7ad48203e56b4f: Status 404 returned error can't find the container with id 60a162e26a0a806cf66d6ecdd527bcc077c677e4209cfb125b7ad48203e56b4f
Nov 03 22:13:27.451969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:27.451935    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf" event={"ID":"1d65b028-da2f-4335-a853-ce90ae9b93a1","Type":"ContainerStarted","Data":"341ac53dd3316f904af603f1295313596176440c14d8b0df5eb42a4c8c116488"}
Nov 03 22:13:27.452419 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:27.451990    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf" event={"ID":"1d65b028-da2f-4335-a853-ce90ae9b93a1","Type":"ContainerStarted","Data":"60a162e26a0a806cf66d6ecdd527bcc077c677e4209cfb125b7ad48203e56b4f"}
Nov 03 22:13:27.597086 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:27.596958    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf" podStartSLOduration=1.596944954 podStartE2EDuration="1.596944954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:27.562741281 +0000 UTC m=+3478.546737867" watchObservedRunningTime="2026-11-03 22:13:27.596944954 +0000 UTC m=+3478.580941548"
Nov 03 22:13:28.459140 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:28.459109    5244 generic.go:356] "Generic (PLEG): container finished" podID="1d65b028-da2f-4335-a853-ce90ae9b93a1" containerID="341ac53dd3316f904af603f1295313596176440c14d8b0df5eb42a4c8c116488" exitCode=0
Nov 03 22:13:28.459597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:28.459174    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf" event={"ID":"1d65b028-da2f-4335-a853-ce90ae9b93a1","Type":"ContainerDied","Data":"341ac53dd3316f904af603f1295313596176440c14d8b0df5eb42a4c8c116488"}
Nov 03 22:13:29.619600 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.619567    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:29.716789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.716750    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca\") pod \"1d65b028-da2f-4335-a853-ce90ae9b93a1\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") "
Nov 03 22:13:29.716983 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.716847    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd\" (UniqueName: \"kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd\") pod \"1d65b028-da2f-4335-a853-ce90ae9b93a1\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") "
Nov 03 22:13:29.716983 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.716887    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca\") pod \"1d65b028-da2f-4335-a853-ce90ae9b93a1\" (UID: \"1d65b028-da2f-4335-a853-ce90ae9b93a1\") "
Nov 03 22:13:29.717296 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.717263    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca" pod "1d65b028-da2f-4335-a853-ce90ae9b93a1" (UID: "1d65b028-da2f-4335-a853-ce90ae9b93a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:13:29.719203 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.719179    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca" pod "1d65b028-da2f-4335-a853-ce90ae9b93a1" (UID: "1d65b028-da2f-4335-a853-ce90ae9b93a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:13:29.719288 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.719216    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd" pod "1d65b028-da2f-4335-a853-ce90ae9b93a1" (UID: "1d65b028-da2f-4335-a853-ce90ae9b93a1"). InnerVolumeSpecName "kube-api-access-z87zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:13:29.818379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.818319    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:13:29.818379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.818374    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87zd\" (UniqueName: \"kubernetes.io/projected/1d65b028-da2f-4335-a853-ce90ae9b93a1-kube-api-access-z87zd\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:13:29.818379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:29.818387    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d65b028-da2f-4335-a853-ce90ae9b93a1-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:13:30.473667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:30.473630    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"
Nov 03 22:13:30.473849 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:30.473667    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf" event={"ID":"1d65b028-da2f-4335-a853-ce90ae9b93a1","Type":"ContainerDied","Data":"60a162e26a0a806cf66d6ecdd527bcc077c677e4209cfb125b7ad48203e56b4f"}
Nov 03 22:13:30.473849 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:30.473718    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a162e26a0a806cf66d6ecdd527bcc077c677e4209cfb125b7ad48203e56b4f"
Nov 03 22:13:37.074273 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.074237    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"]
Nov 03 22:13:37.074708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.074698    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1d65b028-da2f-4335-a853-ce90ae9b93a1" containerName="revoke-keys"
Nov 03 22:13:37.074751 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.074711    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d65b028-da2f-4335-a853-ce90ae9b93a1" containerName="revoke-keys"
Nov 03 22:13:37.074810 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.074799    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d65b028-da2f-4335-a853-ce90ae9b93a1" containerName="revoke-keys"
Nov 03 22:13:37.078756 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.078740    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.134558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.134529    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:13:37.136897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.136874    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:13:37.187615 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.187580    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"]
Nov 03 22:13:37.204902 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.204863    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.205114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.204945    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.205114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.204980    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5r4\" (UniqueName: \"kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.305939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.305897    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.306147 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.305975    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.306147 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.306008    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5r4\" (UniqueName: \"kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.306609 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.306586    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.308574 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.308552    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.364711 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.364629    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5r4\" (UniqueName: \"kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4\") pod \"maas-api-revoke-sub-d3620f1b-vmkkz\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.391433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.391399    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:37.767562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:37.767533    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"]
Nov 03 22:13:37.769963 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:13:37.769937    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c05c0ff_6fd4_4f15_bb9a_2f31f59f4c5c.slice/crio-d74e6b4b4d7a83bcf22f2a0c8d125880a47358394a3edc36ab1b8238117e2ad8 WatchSource:0}: Error finding container d74e6b4b4d7a83bcf22f2a0c8d125880a47358394a3edc36ab1b8238117e2ad8: Status 404 returned error can't find the container with id d74e6b4b4d7a83bcf22f2a0c8d125880a47358394a3edc36ab1b8238117e2ad8
Nov 03 22:13:38.518200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:38.518163    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz" event={"ID":"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c","Type":"ContainerStarted","Data":"cd0dc8ef49fe3db10a3c041143f7831a98d9be45a7322389bcb8fe07b6781a82"}
Nov 03 22:13:38.518657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:38.518222    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz" event={"ID":"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c","Type":"ContainerStarted","Data":"d74e6b4b4d7a83bcf22f2a0c8d125880a47358394a3edc36ab1b8238117e2ad8"}
Nov 03 22:13:38.631778 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:38.631714    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz" podStartSLOduration=1.6316981799999999 podStartE2EDuration="1.63169818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:38.631367742 +0000 UTC m=+3489.615364311" watchObservedRunningTime="2026-11-03 22:13:38.63169818 +0000 UTC m=+3489.615694779"
Nov 03 22:13:39.525443 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:39.525409    5244 generic.go:356] "Generic (PLEG): container finished" podID="4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" containerID="cd0dc8ef49fe3db10a3c041143f7831a98d9be45a7322389bcb8fe07b6781a82" exitCode=0
Nov 03 22:13:39.525904 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:39.525512    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz" event={"ID":"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c","Type":"ContainerDied","Data":"cd0dc8ef49fe3db10a3c041143f7831a98d9be45a7322389bcb8fe07b6781a82"}
Nov 03 22:13:40.674039 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.674013    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:40.745886 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.745854    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4\" (UniqueName: \"kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4\") pod \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") "
Nov 03 22:13:40.746058 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.745924    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca\") pod \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") "
Nov 03 22:13:40.746058 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.745949    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca\") pod \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\" (UID: \"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c\") "
Nov 03 22:13:40.746311 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.746287    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca" pod "4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" (UID: "4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:13:40.748249 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.748226    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca" pod "4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" (UID: "4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:13:40.748322 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.748228    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4" pod "4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" (UID: "4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c"). InnerVolumeSpecName "kube-api-access-2d5r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:13:40.846922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.846828    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5r4\" (UniqueName: \"kubernetes.io/projected/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-kube-api-access-2d5r4\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:13:40.846922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.846861    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:13:40.846922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:40.846876    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:13:41.538019 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:41.537983    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"
Nov 03 22:13:41.538223 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:41.538018    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz" event={"ID":"4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c","Type":"ContainerDied","Data":"d74e6b4b4d7a83bcf22f2a0c8d125880a47358394a3edc36ab1b8238117e2ad8"}
Nov 03 22:13:41.538223 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:41.538079    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74e6b4b4d7a83bcf22f2a0c8d125880a47358394a3edc36ab1b8238117e2ad8"
Nov 03 22:13:41.723946 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:13:41.723915    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:13:52.435857 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:13:52.435817    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:14:22.436877 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:22.436840    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:14:26.329548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.329508    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"]
Nov 03 22:14:26.329970 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.329957    5244 cpu_manager.go:414] "containerMap: removing container" podUID="4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" containerName="revoke-keys"
Nov 03 22:14:26.330014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.329971    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" containerName="revoke-keys"
Nov 03 22:14:26.330060 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.330050    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" containerName="revoke-keys"
Nov 03 22:14:26.333372 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.333355    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.386727 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.386644    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:14:26.390213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.390168    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"]
Nov 03 22:14:26.436151 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.436123    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:14:26.477291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.477254    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.477506 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.477322    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.477506 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.477459    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jvj\" (UniqueName: \"kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.578977 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.578944    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.579201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.579001    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.579201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.579072    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jvj\" (UniqueName: \"kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.580013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.579957    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.581626 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.581605    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.636219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.636177    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jvj\" (UniqueName: \"kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj\") pod \"maas-api-revoke-sub-cfd609ae-gmfzd\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:26.648351 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:26.648301    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:27.027260 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:27.027238    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"]
Nov 03 22:14:27.029708 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:27.029677    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a512b18_492c_4a74_9dd0_6102427a57d3.slice/crio-335880d62ce14ac6e1533a46653db17809e8c21c30455579193d830b6df435ab WatchSource:0}: Error finding container 335880d62ce14ac6e1533a46653db17809e8c21c30455579193d830b6df435ab: Status 404 returned error can't find the container with id 335880d62ce14ac6e1533a46653db17809e8c21c30455579193d830b6df435ab
Nov 03 22:14:27.799806 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:27.799768    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd" event={"ID":"0a512b18-492c-4a74-9dd0-6102427a57d3","Type":"ContainerStarted","Data":"ec003b726ca3cf7c7fb775d151898e38e3b5d44a7321bd737dd9b1fe5754745c"}
Nov 03 22:14:27.800300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:27.799822    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd" event={"ID":"0a512b18-492c-4a74-9dd0-6102427a57d3","Type":"ContainerStarted","Data":"335880d62ce14ac6e1533a46653db17809e8c21c30455579193d830b6df435ab"}
Nov 03 22:14:27.918804 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:27.918752    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd" podStartSLOduration=1.9187386 podStartE2EDuration="1.9187386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:27.916589848 +0000 UTC m=+3538.900586431" watchObservedRunningTime="2026-11-03 22:14:27.9187386 +0000 UTC m=+3538.902735152"
Nov 03 22:14:28.807197 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:28.807164    5244 generic.go:356] "Generic (PLEG): container finished" podID="0a512b18-492c-4a74-9dd0-6102427a57d3" containerID="ec003b726ca3cf7c7fb775d151898e38e3b5d44a7321bd737dd9b1fe5754745c" exitCode=0
Nov 03 22:14:28.807700 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:28.807254    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd" event={"ID":"0a512b18-492c-4a74-9dd0-6102427a57d3","Type":"ContainerDied","Data":"ec003b726ca3cf7c7fb775d151898e38e3b5d44a7321bd737dd9b1fe5754745c"}
Nov 03 22:14:29.945197 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:29.945160    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:30.017971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.017933    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca\") pod \"0a512b18-492c-4a74-9dd0-6102427a57d3\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") "
Nov 03 22:14:30.017971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.017975    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj\" (UniqueName: \"kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj\") pod \"0a512b18-492c-4a74-9dd0-6102427a57d3\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") "
Nov 03 22:14:30.018259 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.018017    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca\") pod \"0a512b18-492c-4a74-9dd0-6102427a57d3\" (UID: \"0a512b18-492c-4a74-9dd0-6102427a57d3\") "
Nov 03 22:14:30.018367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.018351    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca" pod "0a512b18-492c-4a74-9dd0-6102427a57d3" (UID: "0a512b18-492c-4a74-9dd0-6102427a57d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:30.018459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.018442    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:30.020276 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.020240    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj" pod "0a512b18-492c-4a74-9dd0-6102427a57d3" (UID: "0a512b18-492c-4a74-9dd0-6102427a57d3"). InnerVolumeSpecName "kube-api-access-77jvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:30.020418 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.020323    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca" pod "0a512b18-492c-4a74-9dd0-6102427a57d3" (UID: "0a512b18-492c-4a74-9dd0-6102427a57d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:30.119219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.119138    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jvj\" (UniqueName: \"kubernetes.io/projected/0a512b18-492c-4a74-9dd0-6102427a57d3-kube-api-access-77jvj\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:30.119219 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.119170    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a512b18-492c-4a74-9dd0-6102427a57d3-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:30.820080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.820045    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"
Nov 03 22:14:30.820080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.820075    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd" event={"ID":"0a512b18-492c-4a74-9dd0-6102427a57d3","Type":"ContainerDied","Data":"335880d62ce14ac6e1533a46653db17809e8c21c30455579193d830b6df435ab"}
Nov 03 22:14:30.820297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:30.820121    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="335880d62ce14ac6e1533a46653db17809e8c21c30455579193d830b6df435ab"
Nov 03 22:14:31.919966 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:31.919931    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"]
Nov 03 22:14:37.115434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.115397    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"]
Nov 03 22:14:37.115874 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.115859    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0a512b18-492c-4a74-9dd0-6102427a57d3" containerName="revoke-keys"
Nov 03 22:14:37.115921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.115875    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a512b18-492c-4a74-9dd0-6102427a57d3" containerName="revoke-keys"
Nov 03 22:14:37.115973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.115964    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a512b18-492c-4a74-9dd0-6102427a57d3" containerName="revoke-keys"
Nov 03 22:14:37.119287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.119270    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.208036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.207998    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9srr4\" (UniqueName: \"kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.208247 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.208088    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.208247 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.208132    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.222569 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.222531    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:14:37.222760 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.222600    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:14:37.224783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.224762    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"]
Nov 03 22:14:37.309511 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.309476    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9srr4\" (UniqueName: \"kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.309695 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.309537    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.309745 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.309709    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.310216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.310198    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.312348 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.312313    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.366157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.366085    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9srr4\" (UniqueName: \"kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4\") pod \"maas-api-revoke-sub-131c669b-llstx\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.432894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.432866    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:37.475069 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.475018    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"]
Nov 03 22:14:37.603054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.603006    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7682-b71eb1"]
Nov 03 22:14:37.607049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.607024    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:37.668454 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.666742    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8dgqp\""
Nov 03 22:14:37.684038 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.684001    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-b71eb1"]
Nov 03 22:14:37.731538 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.731476    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7bkc\" (UniqueName: \"kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc\") pod \"test-curl-7682-b71eb1\" (UID: \"11512231-dba4-40cd-a47a-e589f106a4e2\") " pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:37.832503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.832461    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7bkc\" (UniqueName: \"kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc\") pod \"test-curl-7682-b71eb1\" (UID: \"11512231-dba4-40cd-a47a-e589f106a4e2\") " pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:37.838756 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.838726    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"]
Nov 03 22:14:37.841054 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:37.841022    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8676dea0_0e70_4c0e_8c2f_83063202f562.slice/crio-85294f817016e0d3c8a876b9308fc2d62672273b1139882f772a0cd740a351be WatchSource:0}: Error finding container 85294f817016e0d3c8a876b9308fc2d62672273b1139882f772a0cd740a351be: Status 404 returned error can't find the container with id 85294f817016e0d3c8a876b9308fc2d62672273b1139882f772a0cd740a351be
Nov 03 22:14:37.870154 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.870113    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx" event={"ID":"8676dea0-0e70-4c0e-8c2f-83063202f562","Type":"ContainerStarted","Data":"85294f817016e0d3c8a876b9308fc2d62672273b1139882f772a0cd740a351be"}
Nov 03 22:14:37.889608 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.889580    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7bkc\" (UniqueName: \"kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc\") pod \"test-curl-7682-b71eb1\" (UID: \"11512231-dba4-40cd-a47a-e589f106a4e2\") " pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:37.920446 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:37.920365    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:38.296046 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:38.296017    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-b71eb1"]
Nov 03 22:14:38.299175 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:38.299147    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11512231_dba4_40cd_a47a_e589f106a4e2.slice/crio-2f4078589420e2c2dca99a06659eef37a278d6a0ae926d8a1b20b48e1fbb9661 WatchSource:0}: Error finding container 2f4078589420e2c2dca99a06659eef37a278d6a0ae926d8a1b20b48e1fbb9661: Status 404 returned error can't find the container with id 2f4078589420e2c2dca99a06659eef37a278d6a0ae926d8a1b20b48e1fbb9661
Nov 03 22:14:38.877316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:38.877282    5244 generic.go:356] "Generic (PLEG): container finished" podID="8676dea0-0e70-4c0e-8c2f-83063202f562" containerID="9ae65c3622ef3883d4f246b0c9419cd1ac53413e6d479b1818356562af3e5005" exitCode=0
Nov 03 22:14:38.877520 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:38.877379    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx" event={"ID":"8676dea0-0e70-4c0e-8c2f-83063202f562","Type":"ContainerDied","Data":"9ae65c3622ef3883d4f246b0c9419cd1ac53413e6d479b1818356562af3e5005"}
Nov 03 22:14:38.879303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:38.879271    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-b71eb1" event={"ID":"11512231-dba4-40cd-a47a-e589f106a4e2","Type":"ContainerStarted","Data":"64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297"}
Nov 03 22:14:38.879469 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:38.879317    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-b71eb1" event={"ID":"11512231-dba4-40cd-a47a-e589f106a4e2","Type":"ContainerStarted","Data":"2f4078589420e2c2dca99a06659eef37a278d6a0ae926d8a1b20b48e1fbb9661"}
Nov 03 22:14:39.101860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:39.101728    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7682-b71eb1" podStartSLOduration=2.101709402 podStartE2EDuration="2.101709402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:39.100080144 +0000 UTC m=+3550.084076703" watchObservedRunningTime="2026-11-03 22:14:39.101709402 +0000 UTC m=+3550.085705962"
Nov 03 22:14:40.036359 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.036313    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:40.158119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.158022    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca\") pod \"8676dea0-0e70-4c0e-8c2f-83063202f562\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") "
Nov 03 22:14:40.158119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.158096    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4\" (UniqueName: \"kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4\") pod \"8676dea0-0e70-4c0e-8c2f-83063202f562\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") "
Nov 03 22:14:40.158408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.158272    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca\") pod \"8676dea0-0e70-4c0e-8c2f-83063202f562\" (UID: \"8676dea0-0e70-4c0e-8c2f-83063202f562\") "
Nov 03 22:14:40.158519 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.158493    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca" pod "8676dea0-0e70-4c0e-8c2f-83063202f562" (UID: "8676dea0-0e70-4c0e-8c2f-83063202f562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:14:40.158763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.158724    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:40.160716 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.160687    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4" pod "8676dea0-0e70-4c0e-8c2f-83063202f562" (UID: "8676dea0-0e70-4c0e-8c2f-83063202f562"). InnerVolumeSpecName "kube-api-access-9srr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:40.160930 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.160896    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca" pod "8676dea0-0e70-4c0e-8c2f-83063202f562" (UID: "8676dea0-0e70-4c0e-8c2f-83063202f562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:40.259961 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.259920    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8676dea0-0e70-4c0e-8c2f-83063202f562-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:40.259961 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.259953    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9srr4\" (UniqueName: \"kubernetes.io/projected/8676dea0-0e70-4c0e-8c2f-83063202f562-kube-api-access-9srr4\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:40.495139 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.495098    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7682-b71eb1"]
Nov 03 22:14:40.498881 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.498848    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7682-b71eb1"]
Nov 03 22:14:40.657786 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.657743    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7682-45f715"]
Nov 03 22:14:40.658260 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.658237    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8676dea0-0e70-4c0e-8c2f-83063202f562" containerName="revoke-keys"
Nov 03 22:14:40.658383 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.658262    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8676dea0-0e70-4c0e-8c2f-83063202f562" containerName="revoke-keys"
Nov 03 22:14:40.658383 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.658299    5244 cpu_manager.go:414] "containerMap: removing container" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" containerName="test-curl-7682-b71eb1"
Nov 03 22:14:40.658383 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.658308    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" containerName="test-curl-7682-b71eb1"
Nov 03 22:14:40.658545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.658490    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8676dea0-0e70-4c0e-8c2f-83063202f562" containerName="revoke-keys"
Nov 03 22:14:40.658545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.658508    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" containerName="test-curl-7682-b71eb1"
Nov 03 22:14:40.661734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.661712    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:40.716467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.716431    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-45f715"]
Nov 03 22:14:40.766121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.766029    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx7p\" (UniqueName: \"kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p\") pod \"test-curl-7682-45f715\" (UID: \"bf2a0cba-1a51-4ec1-855f-d63217dbf918\") " pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:40.866774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.866736    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx7p\" (UniqueName: \"kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p\") pod \"test-curl-7682-45f715\" (UID: \"bf2a0cba-1a51-4ec1-855f-d63217dbf918\") " pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:40.890971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.890943    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"
Nov 03 22:14:40.890971 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.890965    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx" event={"ID":"8676dea0-0e70-4c0e-8c2f-83063202f562","Type":"ContainerDied","Data":"85294f817016e0d3c8a876b9308fc2d62672273b1139882f772a0cd740a351be"}
Nov 03 22:14:40.891222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.891008    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85294f817016e0d3c8a876b9308fc2d62672273b1139882f772a0cd740a351be"
Nov 03 22:14:40.891222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.891015    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7682-b71eb1" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" containerName="test-curl-7682-b71eb1" containerID="cri-o://64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297" gracePeriod=2
Nov 03 22:14:40.923223 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.923186    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx7p\" (UniqueName: \"kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p\") pod \"test-curl-7682-45f715\" (UID: \"bf2a0cba-1a51-4ec1-855f-d63217dbf918\") " pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:40.941806 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:40.941755    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-b71eb1\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:40.974885 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:40.974850    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:40.992574 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:40.992532    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-b71eb1\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:41.347540 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:41.347510    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-45f715"]
Nov 03 22:14:41.349594 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:41.349561    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf2a0cba_1a51_4ec1_855f_d63217dbf918.slice/crio-ecfa2b098a7142e3a87609813f26b258c2dc2e08bb04bf28481613b9ce8bb1cf WatchSource:0}: Error finding container ecfa2b098a7142e3a87609813f26b258c2dc2e08bb04bf28481613b9ce8bb1cf: Status 404 returned error can't find the container with id ecfa2b098a7142e3a87609813f26b258c2dc2e08bb04bf28481613b9ce8bb1cf
Nov 03 22:14:41.897950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:41.897910    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-45f715" event={"ID":"bf2a0cba-1a51-4ec1-855f-d63217dbf918","Type":"ContainerStarted","Data":"a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65"}
Nov 03 22:14:41.897950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:41.897953    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-45f715" event={"ID":"bf2a0cba-1a51-4ec1-855f-d63217dbf918","Type":"ContainerStarted","Data":"ecfa2b098a7142e3a87609813f26b258c2dc2e08bb04bf28481613b9ce8bb1cf"}
Nov 03 22:14:42.012564 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:42.012505    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7682-45f715" podStartSLOduration=2.012490517 podStartE2EDuration="2.012490517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:42.009319117 +0000 UTC m=+3552.993315696" watchObservedRunningTime="2026-11-03 22:14:42.012490517 +0000 UTC m=+3552.996487076"
Nov 03 22:14:43.105610 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.105585    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:43.201149 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.201115    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc\" (UniqueName: \"kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc\") pod \"11512231-dba4-40cd-a47a-e589f106a4e2\" (UID: \"11512231-dba4-40cd-a47a-e589f106a4e2\") "
Nov 03 22:14:43.203483 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.203442    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc" pod "11512231-dba4-40cd-a47a-e589f106a4e2" (UID: "11512231-dba4-40cd-a47a-e589f106a4e2"). InnerVolumeSpecName "kube-api-access-w7bkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:43.302983 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.302947    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7bkc\" (UniqueName: \"kubernetes.io/projected/11512231-dba4-40cd-a47a-e589f106a4e2-kube-api-access-w7bkc\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:43.487667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.487572    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7682-45f715"]
Nov 03 22:14:43.490459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.490428    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7682-45f715"]
Nov 03 22:14:43.642522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.642484    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7682-c67568"]
Nov 03 22:14:43.642988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.642974    5244 cpu_manager.go:414] "containerMap: removing container" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" containerName="test-curl-7682-45f715"
Nov 03 22:14:43.643030 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.642993    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" containerName="test-curl-7682-45f715"
Nov 03 22:14:43.643088 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.643078    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" containerName="test-curl-7682-45f715"
Nov 03 22:14:43.646298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.646282    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:43.699923 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.699882    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-c67568"]
Nov 03 22:14:43.808775 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.808690    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzg7z\" (UniqueName: \"kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z\") pod \"test-curl-7682-c67568\" (UID: \"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb\") " pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:43.910017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.909966    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzg7z\" (UniqueName: \"kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z\") pod \"test-curl-7682-c67568\" (UID: \"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb\") " pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:43.915814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.915780    5244 generic.go:356] "Generic (PLEG): container finished" podID="11512231-dba4-40cd-a47a-e589f106a4e2" containerID="64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297" exitCode=137
Nov 03 22:14:43.916028 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.915936    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7682-45f715" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" containerName="test-curl-7682-45f715" containerID="cri-o://a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65" gracePeriod=2
Nov 03 22:14:43.916189 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.916172    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-b71eb1"
Nov 03 22:14:43.916664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.916642    5244 scope.go:122] "RemoveContainer" containerID="64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297"
Nov 03 22:14:43.928058 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.928017    5244 scope.go:122] "RemoveContainer" containerID="64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297"
Nov 03 22:14:43.928461 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:43.928437    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297\": container with ID starting with 64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297 not found: ID does not exist" containerID="64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297"
Nov 03 22:14:43.928548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.928472    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297"} err="failed to get container status \"64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297\": rpc error: code = NotFound desc = could not find container \"64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297\": container with ID starting with 64823808dd8ac0051027ef6372926e0bb09adc086e8493b1529f43290d1d7297 not found: ID does not exist"
Nov 03 22:14:43.936556 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:43.936529    5244 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11512231_dba4_40cd_a47a_e589f106a4e2.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 22:14:43.966820 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:43.966787    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzg7z\" (UniqueName: \"kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z\") pod \"test-curl-7682-c67568\" (UID: \"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb\") " pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:43.967961 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:43.967935    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-45f715\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:44.018781 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:44.018743    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-45f715\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:44.259736 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:44.259696    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:44.634723 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:44.634697    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-c67568"]
Nov 03 22:14:44.636929 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:44.636900    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddab443ca_c8aa_43f6_a2e8_e76005d0f7eb.slice/crio-6cd437065d8c9852686579c4132b74aaeb659ccc87bde5d60d5575c8cae0224c WatchSource:0}: Error finding container 6cd437065d8c9852686579c4132b74aaeb659ccc87bde5d60d5575c8cae0224c: Status 404 returned error can't find the container with id 6cd437065d8c9852686579c4132b74aaeb659ccc87bde5d60d5575c8cae0224c
Nov 03 22:14:44.927230 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:44.927117    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-c67568" event={"ID":"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb","Type":"ContainerStarted","Data":"461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff"}
Nov 03 22:14:44.927230 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:44.927174    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-c67568" event={"ID":"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb","Type":"ContainerStarted","Data":"6cd437065d8c9852686579c4132b74aaeb659ccc87bde5d60d5575c8cae0224c"}
Nov 03 22:14:45.088592 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:45.088540    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7682-c67568" podStartSLOduration=2.088524585 podStartE2EDuration="2.088524585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:45.087084029 +0000 UTC m=+3556.071080621" watchObservedRunningTime="2026-11-03 22:14:45.088524585 +0000 UTC m=+3556.072521145"
Nov 03 22:14:45.136640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:45.136600    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11512231-dba4-40cd-a47a-e589f106a4e2" path="/var/lib/kubelet/pods/11512231-dba4-40cd-a47a-e589f106a4e2/volumes"
Nov 03 22:14:45.137822 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:45.137788    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-45f715\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:46.169387 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.169362    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:46.334657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.334627    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p\" (UniqueName: \"kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p\") pod \"bf2a0cba-1a51-4ec1-855f-d63217dbf918\" (UID: \"bf2a0cba-1a51-4ec1-855f-d63217dbf918\") "
Nov 03 22:14:46.336990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.336958    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p" pod "bf2a0cba-1a51-4ec1-855f-d63217dbf918" (UID: "bf2a0cba-1a51-4ec1-855f-d63217dbf918"). InnerVolumeSpecName "kube-api-access-2tx7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:46.436487 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.436457    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tx7p\" (UniqueName: \"kubernetes.io/projected/bf2a0cba-1a51-4ec1-855f-d63217dbf918-kube-api-access-2tx7p\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:46.715588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.715534    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7682-c67568"]
Nov 03 22:14:46.718077 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.718049    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7682-c67568"]
Nov 03 22:14:46.866384 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.866350    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7682-1d136e"]
Nov 03 22:14:46.867014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.866993    5244 cpu_manager.go:414] "containerMap: removing container" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" containerName="test-curl-7682-c67568"
Nov 03 22:14:46.867082 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.867018    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" containerName="test-curl-7682-c67568"
Nov 03 22:14:46.867202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.867177    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" containerName="test-curl-7682-c67568"
Nov 03 22:14:46.885748 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.885718    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:46.925747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.925713    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-1d136e"]
Nov 03 22:14:46.940650 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.940614    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" podUID="528e9a14-1d93-4f9b-adab-103d915d056c" containerName="main" containerID="cri-o://a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604" gracePeriod=45
Nov 03 22:14:46.943294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.943260    5244 generic.go:356] "Generic (PLEG): container finished" podID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" containerID="a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65" exitCode=137
Nov 03 22:14:46.943459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.943356    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:46.943459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.943394    5244 scope.go:122] "RemoveContainer" containerID="a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65"
Nov 03 22:14:46.943707 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.943662    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7682-c67568" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" containerName="test-curl-7682-c67568" containerID="cri-o://461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff" gracePeriod=2
Nov 03 22:14:46.953685 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.953660    5244 scope.go:122] "RemoveContainer" containerID="a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65"
Nov 03 22:14:46.954044 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:46.954026    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65\": container with ID starting with a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65 not found: ID does not exist" containerID="a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65"
Nov 03 22:14:46.954122 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:46.954057    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65"} err="failed to get container status \"a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65\": rpc error: code = NotFound desc = could not find container \"a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65\": container with ID starting with a616a3d9d7546618c57666cccd8033c3c4df0ba4d2027c8d414a1062c716ef65 not found: ID does not exist"
Nov 03 22:14:47.043732 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.043639    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmgl\" (UniqueName: \"kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl\") pod \"test-curl-7682-1d136e\" (UID: \"76410920-9867-4ceb-a8e8-ced3f9692595\") " pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:47.046464 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:47.046428    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-c67568\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:47.097099 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:47.097044    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-45f715\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" pod="opendatahub/test-curl-7682-45f715"
Nov 03 22:14:47.140827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.140792    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2a0cba-1a51-4ec1-855f-d63217dbf918" path="/var/lib/kubelet/pods/bf2a0cba-1a51-4ec1-855f-d63217dbf918/volumes"
Nov 03 22:14:47.145106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.145077    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmgl\" (UniqueName: \"kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl\") pod \"test-curl-7682-1d136e\" (UID: \"76410920-9867-4ceb-a8e8-ced3f9692595\") " pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:47.202501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.202473    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmgl\" (UniqueName: \"kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl\") pod \"test-curl-7682-1d136e\" (UID: \"76410920-9867-4ceb-a8e8-ced3f9692595\") " pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:47.218121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.218084    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:14:47.347729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347637    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs\" (UniqueName: \"kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs\") pod \"528e9a14-1d93-4f9b-adab-103d915d056c\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") "
Nov 03 22:14:47.347729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347686    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home\") pod \"528e9a14-1d93-4f9b-adab-103d915d056c\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") "
Nov 03 22:14:47.347729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347706    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir\") pod \"528e9a14-1d93-4f9b-adab-103d915d056c\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") "
Nov 03 22:14:47.348037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347837    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm\") pod \"528e9a14-1d93-4f9b-adab-103d915d056c\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") "
Nov 03 22:14:47.348037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347945    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache\") pod \"528e9a14-1d93-4f9b-adab-103d915d056c\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") "
Nov 03 22:14:47.348037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347958    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home" pod "528e9a14-1d93-4f9b-adab-103d915d056c" (UID: "528e9a14-1d93-4f9b-adab-103d915d056c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:47.348037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.347979    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir" pod "528e9a14-1d93-4f9b-adab-103d915d056c" (UID: "528e9a14-1d93-4f9b-adab-103d915d056c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:47.348253 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.348091    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk\" (UniqueName: \"kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk\") pod \"528e9a14-1d93-4f9b-adab-103d915d056c\" (UID: \"528e9a14-1d93-4f9b-adab-103d915d056c\") "
Nov 03 22:14:47.348253 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.348143    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache" pod "528e9a14-1d93-4f9b-adab-103d915d056c" (UID: "528e9a14-1d93-4f9b-adab-103d915d056c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:47.348591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.348532    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:47.348591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.348559    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:47.348591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.348573    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:47.350216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.350189    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs" pod "528e9a14-1d93-4f9b-adab-103d915d056c" (UID: "528e9a14-1d93-4f9b-adab-103d915d056c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:47.350216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.350208    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk" pod "528e9a14-1d93-4f9b-adab-103d915d056c" (UID: "528e9a14-1d93-4f9b-adab-103d915d056c"). InnerVolumeSpecName "kube-api-access-66mkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:47.350403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.350229    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm" pod "528e9a14-1d93-4f9b-adab-103d915d056c" (UID: "528e9a14-1d93-4f9b-adab-103d915d056c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:47.450072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.450029    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66mkk\" (UniqueName: \"kubernetes.io/projected/528e9a14-1d93-4f9b-adab-103d915d056c-kube-api-access-66mkk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:47.450072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.450063    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528e9a14-1d93-4f9b-adab-103d915d056c-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:47.450072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.450079    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/528e9a14-1d93-4f9b-adab-103d915d056c-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:47.501015 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.500983    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:47.873132 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.872990    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7682-1d136e"]
Nov 03 22:14:47.875402 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:47.875375    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76410920_9867_4ceb_a8e8_ced3f9692595.slice/crio-ab11eb2c34a1b1e81e59c46d96ae0907930554fceba47ca5a451995b243e9a07 WatchSource:0}: Error finding container ab11eb2c34a1b1e81e59c46d96ae0907930554fceba47ca5a451995b243e9a07: Status 404 returned error can't find the container with id ab11eb2c34a1b1e81e59c46d96ae0907930554fceba47ca5a451995b243e9a07
Nov 03 22:14:47.951818 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.951779    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-1d136e" event={"ID":"76410920-9867-4ceb-a8e8-ced3f9692595","Type":"ContainerStarted","Data":"6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449"}
Nov 03 22:14:47.951818 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.951822    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7682-1d136e" event={"ID":"76410920-9867-4ceb-a8e8-ced3f9692595","Type":"ContainerStarted","Data":"ab11eb2c34a1b1e81e59c46d96ae0907930554fceba47ca5a451995b243e9a07"}
Nov 03 22:14:47.953207 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.953174    5244 generic.go:356] "Generic (PLEG): container finished" podID="528e9a14-1d93-4f9b-adab-103d915d056c" containerID="a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604" exitCode=0
Nov 03 22:14:47.953367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.953214    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"
Nov 03 22:14:47.953367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.953215    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" event={"ID":"528e9a14-1d93-4f9b-adab-103d915d056c","Type":"ContainerDied","Data":"a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604"}
Nov 03 22:14:47.953367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.953350    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" event={"ID":"528e9a14-1d93-4f9b-adab-103d915d056c","Type":"ContainerDied","Data":"04187e6e5cbd5da5992a4f783ed3469fb34ba7b4e9cadb10ab834676985cb165"}
Nov 03 22:14:47.953528 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.953378    5244 scope.go:122] "RemoveContainer" containerID="a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604"
Nov 03 22:14:47.971963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.971941    5244 scope.go:122] "RemoveContainer" containerID="a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604"
Nov 03 22:14:47.972370 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:47.972319    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604\": container with ID starting with a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604 not found: ID does not exist" containerID="a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604"
Nov 03 22:14:47.972415 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:47.972383    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604"} err="failed to get container status \"a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604\": rpc error: code = NotFound desc = could not find container \"a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604\": container with ID starting with a8592a5d017b165dbe48df8314f6efab94bb6ac4ad4aa7f74fdcbaadbc170604 not found: ID does not exist"
Nov 03 22:14:48.174351 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:48.174198    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7682-1d136e" podStartSLOduration=2.174176922 podStartE2EDuration="2.174176922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:48.063573232 +0000 UTC m=+3559.047569792" watchObservedRunningTime="2026-11-03 22:14:48.174176922 +0000 UTC m=+3559.158173482"
Nov 03 22:14:48.174863 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:48.174839    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"]
Nov 03 22:14:48.177454 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:48.177432    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg"]
Nov 03 22:14:48.179445 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:48.179417    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-29dbd9/auth-test-model-29dbd9-kserve-6fbcfb4869-qkgbg" podUID="528e9a14-1d93-4f9b-adab-103d915d056c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": context deadline exceeded"
Nov 03 22:14:49.138535 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.138502    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528e9a14-1d93-4f9b-adab-103d915d056c" path="/var/lib/kubelet/pods/528e9a14-1d93-4f9b-adab-103d915d056c/volumes"
Nov 03 22:14:49.152815 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.152791    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:49.203608 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:49.203518    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-c67568\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:49.269123 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.269092    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z\" (UniqueName: \"kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z\") pod \"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb\" (UID: \"dab443ca-c8aa-43f6-a2e8-e76005d0f7eb\") "
Nov 03 22:14:49.271393 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.271364    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z" pod "dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" (UID: "dab443ca-c8aa-43f6-a2e8-e76005d0f7eb"). InnerVolumeSpecName "kube-api-access-tzg7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:49.370138 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.370104    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzg7z\" (UniqueName: \"kubernetes.io/projected/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb-kube-api-access-tzg7z\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:49.543757 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.543669    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7682-1d136e"]
Nov 03 22:14:49.547950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.547922    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7682-1d136e"]
Nov 03 22:14:49.967136 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.967093    5244 generic.go:356] "Generic (PLEG): container finished" podID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" containerID="461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff" exitCode=137
Nov 03 22:14:49.967316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.967150    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:49.967316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.967178    5244 scope.go:122] "RemoveContainer" containerID="461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff"
Nov 03 22:14:49.967479 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.967456    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7682-1d136e" podUID="76410920-9867-4ceb-a8e8-ced3f9692595" containerName="test-curl-7682-1d136e" containerID="cri-o://6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449" gracePeriod=2
Nov 03 22:14:49.979388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.979360    5244 scope.go:122] "RemoveContainer" containerID="461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff"
Nov 03 22:14:49.979711 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:49.979687    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff\": container with ID starting with 461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff not found: ID does not exist" containerID="461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff"
Nov 03 22:14:49.979783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:49.979720    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff"} err="failed to get container status \"461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff\": rpc error: code = NotFound desc = could not find container \"461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff\": container with ID starting with 461ec2a9f476a172d1b660fdeebfce50ce18b08b7802cbe3c28a67a51b2188ff not found: ID does not exist"
Nov 03 22:14:50.017837 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:50.017804    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-c67568\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:50.068254 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:50.068213    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-c67568\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" pod="opendatahub/test-curl-7682-c67568"
Nov 03 22:14:50.118996 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:50.118955    5244 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7682-1d136e\" is forbidden: User \"system:node:ip-10-0-129-65.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.us-west-1.compute.internal' and this object" podUID="76410920-9867-4ceb-a8e8-ced3f9692595" pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:51.136753 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.136673    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab443ca-c8aa-43f6-a2e8-e76005d0f7eb" path="/var/lib/kubelet/pods/dab443ca-c8aa-43f6-a2e8-e76005d0f7eb/volumes"
Nov 03 22:14:51.669596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.669561    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"]
Nov 03 22:14:51.670030 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.670016    5244 cpu_manager.go:414] "containerMap: removing container" podUID="528e9a14-1d93-4f9b-adab-103d915d056c" containerName="main"
Nov 03 22:14:51.670075 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.670034    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528e9a14-1d93-4f9b-adab-103d915d056c" containerName="main"
Nov 03 22:14:51.670075 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.670064    5244 cpu_manager.go:414] "containerMap: removing container" podUID="76410920-9867-4ceb-a8e8-ced3f9692595" containerName="test-curl-7682-1d136e"
Nov 03 22:14:51.670075 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.670069    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76410920-9867-4ceb-a8e8-ced3f9692595" containerName="test-curl-7682-1d136e"
Nov 03 22:14:51.670173 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.670144    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="528e9a14-1d93-4f9b-adab-103d915d056c" containerName="main"
Nov 03 22:14:51.670173 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.670157    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="76410920-9867-4ceb-a8e8-ced3f9692595" containerName="test-curl-7682-1d136e"
Nov 03 22:14:51.674788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.674771    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.708367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.699802    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.708745 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708696    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.708889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708767    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.708889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708808    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.708889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708840    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.708889 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708878    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.709087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708914    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.709087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.708964    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.709087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.709060    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxgn\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.780214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.780126    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"]
Nov 03 22:14:51.809822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809786    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809838    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809866    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809893    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809919    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809939    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.809966    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.810019    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxgn\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.810069    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810380 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.810236    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810561 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.810415    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.810562    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.810682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.810625    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.811048 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.811026    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.813268 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.813240    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.813408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.813343    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.826643 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.826607    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-288188-openshift-default-dockercfg-jx5t5\""
Nov 03 22:14:51.917389 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.917041    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:51.918621 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.918593    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxgn\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn\") pod \"e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") " pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:52.000475 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:51.994631    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:52.205072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.205045    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:52.315640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.315543    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl\" (UniqueName: \"kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl\") pod \"76410920-9867-4ceb-a8e8-ced3f9692595\" (UID: \"76410920-9867-4ceb-a8e8-ced3f9692595\") "
Nov 03 22:14:52.317691 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.317668    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl" pod "76410920-9867-4ceb-a8e8-ced3f9692595" (UID: "76410920-9867-4ceb-a8e8-ced3f9692595"). InnerVolumeSpecName "kube-api-access-sxmgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:52.417184 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.417139    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmgl\" (UniqueName: \"kubernetes.io/projected/76410920-9867-4ceb-a8e8-ced3f9692595-kube-api-access-sxmgl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:52.439463 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:52.439420    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:14:52.460673 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.460645    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"]
Nov 03 22:14:52.462953 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:14:52.462923    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e008d8_348f_4b8a_9f7a_23066154dad2.slice/crio-6aca8c4751e871d33dde3f8a0060fee91f0a52d0865756709a4c72785e781f02 WatchSource:0}: Error finding container 6aca8c4751e871d33dde3f8a0060fee91f0a52d0865756709a4c72785e781f02: Status 404 returned error can't find the container with id 6aca8c4751e871d33dde3f8a0060fee91f0a52d0865756709a4c72785e781f02
Nov 03 22:14:52.466351 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.466297    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:14:52.466453 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.466395    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:14:52.466453 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.466432    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:14:52.893232 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.893184    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" podUID="64ce8a87-1937-4c86-959f-1edf618feefe" containerName="main" containerID="cri-o://f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53" gracePeriod=45
Nov 03 22:14:52.989796 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.989765    5244 generic.go:356] "Generic (PLEG): container finished" podID="76410920-9867-4ceb-a8e8-ced3f9692595" containerID="6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449" exitCode=137
Nov 03 22:14:52.990001 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.989827    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7682-1d136e"
Nov 03 22:14:52.990001 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.989905    5244 scope.go:122] "RemoveContainer" containerID="6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449"
Nov 03 22:14:52.991927 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.991901    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" event={"ID":"21e008d8-348f-4b8a-9f7a-23066154dad2","Type":"ContainerStarted","Data":"7f5afe69b2abc43ba5a9cc426c4f6583af90dc9905895091a1d486a3d84fed72"}
Nov 03 22:14:52.992048 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.991948    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" event={"ID":"21e008d8-348f-4b8a-9f7a-23066154dad2","Type":"ContainerStarted","Data":"6aca8c4751e871d33dde3f8a0060fee91f0a52d0865756709a4c72785e781f02"}
Nov 03 22:14:52.998988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:52.998961    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:53.007782 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.007753    5244 scope.go:122] "RemoveContainer" containerID="6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449"
Nov 03 22:14:53.008149 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:53.008124    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449\": container with ID starting with 6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449 not found: ID does not exist" containerID="6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449"
Nov 03 22:14:53.008239 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.008161    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449"} err="failed to get container status \"6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449\": rpc error: code = NotFound desc = could not find container \"6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449\": container with ID starting with 6aff9cc7c0ee6948c0c8099f11127f59b743108c9f4896f7c7b6b8b321a11449 not found: ID does not exist"
Nov 03 22:14:53.056825 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.056789    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" podUID="64ce8a87-1937-4c86-959f-1edf618feefe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": dial tcp 10.133.0.153:8000: connect: connection refused"
Nov 03 22:14:53.147358 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.147290    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76410920-9867-4ceb-a8e8-ced3f9692595" path="/var/lib/kubelet/pods/76410920-9867-4ceb-a8e8-ced3f9692595/volumes"
Nov 03 22:14:53.169882 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.169844    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:14:53.228037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228002    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache\") pod \"64ce8a87-1937-4c86-959f-1edf618feefe\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") "
Nov 03 22:14:53.228555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228101    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm\") pod \"64ce8a87-1937-4c86-959f-1edf618feefe\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") "
Nov 03 22:14:53.228555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228147    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs\" (UniqueName: \"kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs\") pod \"64ce8a87-1937-4c86-959f-1edf618feefe\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") "
Nov 03 22:14:53.228555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228188    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl\" (UniqueName: \"kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl\") pod \"64ce8a87-1937-4c86-959f-1edf618feefe\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") "
Nov 03 22:14:53.228555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228370    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache" pod "64ce8a87-1937-4c86-959f-1edf618feefe" (UID: "64ce8a87-1937-4c86-959f-1edf618feefe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:53.228555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228381    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir\") pod \"64ce8a87-1937-4c86-959f-1edf618feefe\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") "
Nov 03 22:14:53.228555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228488    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home\") pod \"64ce8a87-1937-4c86-959f-1edf618feefe\" (UID: \"64ce8a87-1937-4c86-959f-1edf618feefe\") "
Nov 03 22:14:53.228891 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228696    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir" pod "64ce8a87-1937-4c86-959f-1edf618feefe" (UID: "64ce8a87-1937-4c86-959f-1edf618feefe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:53.229056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.228987    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:53.229056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.229015    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:53.229056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.229017    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home" pod "64ce8a87-1937-4c86-959f-1edf618feefe" (UID: "64ce8a87-1937-4c86-959f-1edf618feefe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:53.230650 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.230622    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm" pod "64ce8a87-1937-4c86-959f-1edf618feefe" (UID: "64ce8a87-1937-4c86-959f-1edf618feefe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:14:53.231057 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.231038    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs" pod "64ce8a87-1937-4c86-959f-1edf618feefe" (UID: "64ce8a87-1937-4c86-959f-1edf618feefe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:14:53.231146 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.231134    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl" pod "64ce8a87-1937-4c86-959f-1edf618feefe" (UID: "64ce8a87-1937-4c86-959f-1edf618feefe"). InnerVolumeSpecName "kube-api-access-nqlpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:14:53.309560 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.309481    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podStartSLOduration=2.309458205 podStartE2EDuration="2.309458205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:53.195555849 +0000 UTC m=+3564.179552427" watchObservedRunningTime="2026-11-03 22:14:53.309458205 +0000 UTC m=+3564.293454762"
Nov 03 22:14:53.330603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.330561    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqlpl\" (UniqueName: \"kubernetes.io/projected/64ce8a87-1937-4c86-959f-1edf618feefe-kube-api-access-nqlpl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:53.330603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.330605    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:53.330873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.330620    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64ce8a87-1937-4c86-959f-1edf618feefe-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:53.330873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:53.330635    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ce8a87-1937-4c86-959f-1edf618feefe-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:14:54.000240 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.000202    5244 generic.go:356] "Generic (PLEG): container finished" podID="64ce8a87-1937-4c86-959f-1edf618feefe" containerID="f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53" exitCode=0
Nov 03 22:14:54.000485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.000366    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" event={"ID":"64ce8a87-1937-4c86-959f-1edf618feefe","Type":"ContainerDied","Data":"f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53"}
Nov 03 22:14:54.000485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.000419    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz" event={"ID":"64ce8a87-1937-4c86-959f-1edf618feefe","Type":"ContainerDied","Data":"760027d6fdaae090c96983c550b50a93780778b532ebcef02c500c2150566417"}
Nov 03 22:14:54.000485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.000441    5244 scope.go:122] "RemoveContainer" containerID="f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53"
Nov 03 22:14:54.000485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.000378    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"
Nov 03 22:14:54.007603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.007581    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:54.012322 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.012305    5244 scope.go:122] "RemoveContainer" containerID="f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53"
Nov 03 22:14:54.012709 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:14:54.012687    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53\": container with ID starting with f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53 not found: ID does not exist" containerID="f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53"
Nov 03 22:14:54.012784 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.012720    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53"} err="failed to get container status \"f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53\": rpc error: code = NotFound desc = could not find container \"f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53\": container with ID starting with f0bd1884824695764462494e23f41748668bc1aaa33854d23fe6bf801983ce53 not found: ID does not exist"
Nov 03 22:14:54.280173 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.280081    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"]
Nov 03 22:14:54.283832 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:54.283793    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-0d45ba/auth-test-model-0d45ba-kserve-5dcf6b849c-kjdzz"]
Nov 03 22:14:55.007557 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:55.007523    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:55.008683 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:55.008666    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:14:55.136728 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:55.136699    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ce8a87-1937-4c86-959f-1edf618feefe" path="/var/lib/kubelet/pods/64ce8a87-1937-4c86-959f-1edf618feefe/volumes"
Nov 03 22:14:59.710437 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.710405    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"]
Nov 03 22:14:59.710905 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.710857    5244 cpu_manager.go:414] "containerMap: removing container" podUID="64ce8a87-1937-4c86-959f-1edf618feefe" containerName="main"
Nov 03 22:14:59.710905 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.710868    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ce8a87-1937-4c86-959f-1edf618feefe" containerName="main"
Nov 03 22:14:59.710980 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.710968    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ce8a87-1937-4c86-959f-1edf618feefe" containerName="main"
Nov 03 22:14:59.714058 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.714041    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:14:59.771977 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.771943    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"]
Nov 03 22:14:59.810431 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.810379    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp95q\" (UniqueName: \"kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q\") pod \"payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:14:59.810667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.810636    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:14:59.832615 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.832576    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-288188\""
Nov 03 22:14:59.833244 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.833214    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-288188-dockercfg-786k2\""
Nov 03 22:14:59.860906 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.860867    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"]
Nov 03 22:14:59.865743 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.865712    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:14:59.911459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.911429    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp95q\" (UniqueName: \"kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q\") pod \"payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:14:59.911645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.911564    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:14:59.912496 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.912475    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:14:59.917420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.917390    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"]
Nov 03 22:14:59.968087 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:14:59.968013    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp95q\" (UniqueName: \"kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q\") pod \"payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:15:00.012939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.012909    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-288188-849858c758-9nctt\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.013158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.013092    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfmr\" (UniqueName: \"kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr\") pod \"payload-processing-e2e-extmod-a-288188-849858c758-9nctt\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.027610 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.027580    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:15:00.114593 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.114563    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-288188-849858c758-9nctt\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.114771 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.114670    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfmr\" (UniqueName: \"kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr\") pod \"payload-processing-e2e-extmod-a-288188-849858c758-9nctt\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.115501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.115472    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-288188-849858c758-9nctt\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.158053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.158017    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"]
Nov 03 22:15:00.163184 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.163158    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"]
Nov 03 22:15:00.163374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.163286    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.167921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.167899    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"]
Nov 03 22:15:00.168052 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.168030    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.171701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.171678    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"]
Nov 03 22:15:00.171821 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.171795    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.179615 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.179585    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfmr\" (UniqueName: \"kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr\") pod \"payload-processing-e2e-extmod-a-288188-849858c758-9nctt\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.185569 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.185542    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.185908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.185892    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"]
Nov 03 22:15:00.190684 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.190460    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.190855 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.190833    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"]
Nov 03 22:15:00.190942 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.190925    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:00.195454 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.195431    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.208467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.208441    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"]
Nov 03 22:15:00.213387 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.213354    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"]
Nov 03 22:15:00.213499 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.213429    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.219006 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.218892    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:15:00.219235 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.219209    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.219409 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.219389    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskbj\" (UniqueName: \"kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.225277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.224734    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"]
Nov 03 22:15:00.225277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.224776    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"]
Nov 03 22:15:00.225277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.224874    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.225877 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.225857    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"]
Nov 03 22:15:00.290088 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.289883    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"]
Nov 03 22:15:00.290088 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.289959    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"]
Nov 03 22:15:00.294403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.294379    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"]
Nov 03 22:15:00.297591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.297566    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"]
Nov 03 22:15:00.321224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321193    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.321407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321241    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.321407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321263    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkvj\" (UniqueName: \"kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.321407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321396    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7dw\" (UniqueName: \"kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.321580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321454    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.321580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321487    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56fb\" (UniqueName: \"kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.321580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321515    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.321580 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321553    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.321777 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321596    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlzq\" (UniqueName: \"kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.321777 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321662    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbgx\" (UniqueName: \"kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.321777 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321699    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.321944 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321861    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n69n\" (UniqueName: \"kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.321944 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321924    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.322054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.321981    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrck\" (UniqueName: \"kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.322054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.322043    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskbj\" (UniqueName: \"kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.322269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.322250    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.322442 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.322416    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.330829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.330801    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"]
Nov 03 22:15:00.378604 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.378571    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskbj\" (UniqueName: \"kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.411370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.411342    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"]
Nov 03 22:15:00.413141 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:00.413111    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7ace1d_a569_46e7_bc8a_36428bc956f6.slice/crio-efd91745b4a062dff23c02105dfe5eab78cec52bdbf5386cfd7c6d6f9f9069b0 WatchSource:0}: Error finding container efd91745b4a062dff23c02105dfe5eab78cec52bdbf5386cfd7c6d6f9f9069b0: Status 404 returned error can't find the container with id efd91745b4a062dff23c02105dfe5eab78cec52bdbf5386cfd7c6d6f9f9069b0
Nov 03 22:15:00.423294 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423270    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.423406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423345    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.423472 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423449    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.423516 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423494    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkvj\" (UniqueName: \"kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.423575 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423523    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7dw\" (UniqueName: \"kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.423575 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423564    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56fb\" (UniqueName: \"kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.423682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423584    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.423682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423624    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.423682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423669    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlzq\" (UniqueName: \"kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.423836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423730    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbgx\" (UniqueName: \"kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.423836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423773    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.423836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423798    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n69n\" (UniqueName: \"kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.423836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.423831    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.424073 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424037    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrck\" (UniqueName: \"kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.424191 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424098    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.424191 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424122    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.424302 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424221    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.424555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424523    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.424673 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424592    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.424763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424744    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.424817 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.424781    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.479953 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.479921    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7dw\" (UniqueName: \"kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.480216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.480189    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkvj\" (UniqueName: \"kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.480458 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.480441    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56fb\" (UniqueName: \"kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.485216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.485192    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:00.524019 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.523990    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n69n\" (UniqueName: \"kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n\") pod \"maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.524221 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.524071    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlzq\" (UniqueName: \"kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq\") pod \"maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.524314 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.524233    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrck\" (UniqueName: \"kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.525340 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.525308    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:00.530206 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.529948    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbgx\" (UniqueName: \"kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx\") pod \"maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.543621 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.543474    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:00.559594 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.559570    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:00.596222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.596187    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"]
Nov 03 22:15:00.604501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.604476    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:00.617064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.614696    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:00.640083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.629557    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:00.640083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.639733    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:00.868914 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.868889    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"]
Nov 03 22:15:00.871776 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:00.871754    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499fbb98_6d07_4117_8dd1_99178f6fbcc3.slice/crio-bdcc6267bd2707ce8e73ea0a7b2ccbaec95cb3c0de6ca0d83b1236e222a95a3d WatchSource:0}: Error finding container bdcc6267bd2707ce8e73ea0a7b2ccbaec95cb3c0de6ca0d83b1236e222a95a3d: Status 404 returned error can't find the container with id bdcc6267bd2707ce8e73ea0a7b2ccbaec95cb3c0de6ca0d83b1236e222a95a3d
Nov 03 22:15:00.941791 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.941759    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"]
Nov 03 22:15:00.943679 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:00.943645    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee72c2b_3938_457a_98a3_565450f0c7ed.slice/crio-918f1a97147ea9e172f2a5c62e01b22733c1454c0e9275231a6aa20ceefcedfb WatchSource:0}: Error finding container 918f1a97147ea9e172f2a5c62e01b22733c1454c0e9275231a6aa20ceefcedfb: Status 404 returned error can't find the container with id 918f1a97147ea9e172f2a5c62e01b22733c1454c0e9275231a6aa20ceefcedfb
Nov 03 22:15:00.989993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.989916    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"]
Nov 03 22:15:00.992300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:00.992277    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"]
Nov 03 22:15:00.994449 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:00.994422    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50da3769_6de9_4b39_b217_3da2b24d98ff.slice/crio-1f83c1d072feb03deed0ff9c951f772fd25ed77816b31043d4aa7eb5379421af WatchSource:0}: Error finding container 1f83c1d072feb03deed0ff9c951f772fd25ed77816b31043d4aa7eb5379421af: Status 404 returned error can't find the container with id 1f83c1d072feb03deed0ff9c951f772fd25ed77816b31043d4aa7eb5379421af
Nov 03 22:15:00.995219 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:00.995197    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a77b83c_515e_415a_b375_0c72b0abfb85.slice/crio-993efcf917fa1058502e709a9a4594d76f8561b388a2101a457b0da4d11a4fc9 WatchSource:0}: Error finding container 993efcf917fa1058502e709a9a4594d76f8561b388a2101a457b0da4d11a4fc9: Status 404 returned error can't find the container with id 993efcf917fa1058502e709a9a4594d76f8561b388a2101a457b0da4d11a4fc9
Nov 03 22:15:01.047848 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.043572    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"]
Nov 03 22:15:01.048022 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.047988    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" event={"ID":"dee72c2b-3938-457a-98a3-565450f0c7ed","Type":"ContainerStarted","Data":"918f1a97147ea9e172f2a5c62e01b22733c1454c0e9275231a6aa20ceefcedfb"}
Nov 03 22:15:01.049871 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.049846    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk" event={"ID":"6d7ace1d-a569-46e7-bc8a-36428bc956f6","Type":"ContainerStarted","Data":"703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660"}
Nov 03 22:15:01.049973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.049899    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk" event={"ID":"6d7ace1d-a569-46e7-bc8a-36428bc956f6","Type":"ContainerStarted","Data":"efd91745b4a062dff23c02105dfe5eab78cec52bdbf5386cfd7c6d6f9f9069b0"}
Nov 03 22:15:01.049973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.049929    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:15:01.051697 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:01.051658    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbe2af4_e992_497b_a95a_d672a29c842e.slice/crio-22dbd0655d9d730ffd6a551e3826a007b274b263b3043cd89d0a067f03cb53e1 WatchSource:0}: Error finding container 22dbd0655d9d730ffd6a551e3826a007b274b263b3043cd89d0a067f03cb53e1: Status 404 returned error can't find the container with id 22dbd0655d9d730ffd6a551e3826a007b274b263b3043cd89d0a067f03cb53e1
Nov 03 22:15:01.051992 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.051966    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerStarted","Data":"1f83c1d072feb03deed0ff9c951f772fd25ed77816b31043d4aa7eb5379421af"}
Nov 03 22:15:01.054008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.053820    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerStarted","Data":"993efcf917fa1058502e709a9a4594d76f8561b388a2101a457b0da4d11a4fc9"}
Nov 03 22:15:01.056522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.056493    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" event={"ID":"499fbb98-6d07-4117-8dd1-99178f6fbcc3","Type":"ContainerStarted","Data":"bdcc6267bd2707ce8e73ea0a7b2ccbaec95cb3c0de6ca0d83b1236e222a95a3d"}
Nov 03 22:15:01.058403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.058379    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt" event={"ID":"76ed667a-8f47-4f86-9a40-5c2b1db7600d","Type":"ContainerStarted","Data":"1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0"}
Nov 03 22:15:01.058510 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.058420    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt" event={"ID":"76ed667a-8f47-4f86-9a40-5c2b1db7600d","Type":"ContainerStarted","Data":"88d3c9fca5193681319c5054d0a0b7f345c05962bbf1fcd8f66070c6f11c19f1"}
Nov 03 22:15:01.058578 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.058546    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:01.095707 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.095681    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"]
Nov 03 22:15:01.097278 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:01.097250    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2652d3bb_1745_4a5e_8921_c84792ec9f34.slice/crio-856a6aede35265b5d6f3db22ec5a641f21b7d1352e87b1f1510c6e075bc796be WatchSource:0}: Error finding container 856a6aede35265b5d6f3db22ec5a641f21b7d1352e87b1f1510c6e075bc796be: Status 404 returned error can't find the container with id 856a6aede35265b5d6f3db22ec5a641f21b7d1352e87b1f1510c6e075bc796be
Nov 03 22:15:01.146514 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.145966    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"]
Nov 03 22:15:01.149712 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.149687    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"]
Nov 03 22:15:01.239485 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:01.239442    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6fb05c_594a_4e9a_bf73_e241622d52e0.slice/crio-91c745f5f6694a5ddb9aba66695c2da229b88436329d25124790a3b8a2a838ed WatchSource:0}: Error finding container 91c745f5f6694a5ddb9aba66695c2da229b88436329d25124790a3b8a2a838ed: Status 404 returned error can't find the container with id 91c745f5f6694a5ddb9aba66695c2da229b88436329d25124790a3b8a2a838ed
Nov 03 22:15:01.243172 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:01.242789    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09a28fe_6665_44c3_90fa_c3dc9035b301.slice/crio-9afb02f42cf2c8783664a445ce7903edad59de7c2dc32f22fcfaa4535913b438 WatchSource:0}: Error finding container 9afb02f42cf2c8783664a445ce7903edad59de7c2dc32f22fcfaa4535913b438: Status 404 returned error can't find the container with id 9afb02f42cf2c8783664a445ce7903edad59de7c2dc32f22fcfaa4535913b438
Nov 03 22:15:01.369557 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:01.369375    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt" podStartSLOduration=2.369354865 podStartE2EDuration="2.369354865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:01.254141586 +0000 UTC m=+3572.238138147" watchObservedRunningTime="2026-11-03 22:15:01.369354865 +0000 UTC m=+3572.353351424"
Nov 03 22:15:02.069052 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.069019    5244 generic.go:356] "Generic (PLEG): container finished" podID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerID="e22f2a0bb15b947c5318dbf4a56b8ac9ecfe61aa5e6723f609cb5cf1acab9a86" exitCode=60
Nov 03 22:15:02.069684 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.069225    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" event={"ID":"499fbb98-6d07-4117-8dd1-99178f6fbcc3","Type":"ContainerDied","Data":"e22f2a0bb15b947c5318dbf4a56b8ac9ecfe61aa5e6723f609cb5cf1acab9a86"}
Nov 03 22:15:02.069784 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.069691    5244 scope.go:122] "RemoveContainer" containerID="e22f2a0bb15b947c5318dbf4a56b8ac9ecfe61aa5e6723f609cb5cf1acab9a86"
Nov 03 22:15:02.075993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.072549    5244 generic.go:356] "Generic (PLEG): container finished" podID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerID="971f311bde8e526454c76bb31e449bdfa61015d0bea2d98ec544907c475f71db" exitCode=60
Nov 03 22:15:02.075993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.072634    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" event={"ID":"dee72c2b-3938-457a-98a3-565450f0c7ed","Type":"ContainerDied","Data":"971f311bde8e526454c76bb31e449bdfa61015d0bea2d98ec544907c475f71db"}
Nov 03 22:15:02.075993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.072984    5244 scope.go:122] "RemoveContainer" containerID="971f311bde8e526454c76bb31e449bdfa61015d0bea2d98ec544907c475f71db"
Nov 03 22:15:02.081101 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.081072    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerStarted","Data":"0ba0b8cccf9c023e5117be9a637f82de35213c33cd8efe004f2737935d51ffcf"}
Nov 03 22:15:02.081246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.081129    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerStarted","Data":"856a6aede35265b5d6f3db22ec5a641f21b7d1352e87b1f1510c6e075bc796be"}
Nov 03 22:15:02.083281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.082798    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerStarted","Data":"3dd57394508274b54fc3c951823977abadc787da7dfa2e090355cc9103107137"}
Nov 03 22:15:02.083281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.082843    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerStarted","Data":"22dbd0655d9d730ffd6a551e3826a007b274b263b3043cd89d0a067f03cb53e1"}
Nov 03 22:15:02.085050 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.085028    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerStarted","Data":"f51f35ddda0f734fd1504d9f116419d05d55cb6d329295fbafc0490cd152f36f"}
Nov 03 22:15:02.087371 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.087311    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerStarted","Data":"6471237c99cad1a8db7222149686ef77f8d8fed21390a5102b082d606f092d20"}
Nov 03 22:15:02.087479 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.087382    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerStarted","Data":"91c745f5f6694a5ddb9aba66695c2da229b88436329d25124790a3b8a2a838ed"}
Nov 03 22:15:02.089142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.089119    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerStarted","Data":"97e9eda8f9bc19825f80ba83467cf5cfb8738932cbefd9abd35dc09402595b44"}
Nov 03 22:15:02.091179 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.091158    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerStarted","Data":"03b8007e3dc13ef2476bb3fac989439fd933edf12a00bc6b4bed31ce89f81053"}
Nov 03 22:15:02.091402 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.091366    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerStarted","Data":"9afb02f42cf2c8783664a445ce7903edad59de7c2dc32f22fcfaa4535913b438"}
Nov 03 22:15:02.183871 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.183805    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk" podStartSLOduration=3.18378774 podStartE2EDuration="3.18378774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:01.370597915 +0000 UTC m=+3572.354594476" watchObservedRunningTime="2026-11-03 22:15:02.18378774 +0000 UTC m=+3573.167784298"
Nov 03 22:15:02.294147 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.294090    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" podStartSLOduration=2.294075482 podStartE2EDuration="2.294075482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:02.292050753 +0000 UTC m=+3573.276047312" watchObservedRunningTime="2026-11-03 22:15:02.294075482 +0000 UTC m=+3573.278072040"
Nov 03 22:15:02.404271 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.404219    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" podStartSLOduration=2.404204943 podStartE2EDuration="2.404204943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:02.403264035 +0000 UTC m=+3573.387260616" watchObservedRunningTime="2026-11-03 22:15:02.404204943 +0000 UTC m=+3573.388201499"
Nov 03 22:15:02.646548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.646488    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" podStartSLOduration=2.646472698 podStartE2EDuration="2.646472698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:02.642505929 +0000 UTC m=+3573.626502491" watchObservedRunningTime="2026-11-03 22:15:02.646472698 +0000 UTC m=+3573.630469256"
Nov 03 22:15:02.755207 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.755145    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" podStartSLOduration=2.755126549 podStartE2EDuration="2.755126549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:02.753281712 +0000 UTC m=+3573.737278292" watchObservedRunningTime="2026-11-03 22:15:02.755126549 +0000 UTC m=+3573.739123143"
Nov 03 22:15:02.866634 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.866519    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" podStartSLOduration=2.866504581 podStartE2EDuration="2.866504581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:02.863887899 +0000 UTC m=+3573.847884499" watchObservedRunningTime="2026-11-03 22:15:02.866504581 +0000 UTC m=+3573.850501140"
Nov 03 22:15:02.978343 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:02.978258    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" podStartSLOduration=2.978238982 podStartE2EDuration="2.978238982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:02.976620403 +0000 UTC m=+3573.960616980" watchObservedRunningTime="2026-11-03 22:15:02.978238982 +0000 UTC m=+3573.962235542"
Nov 03 22:15:03.100436 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.100405    5244 generic.go:356] "Generic (PLEG): container finished" podID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerID="06ffae7bd575b7c0116765d1d3b67f08c08214ef9ee97f18807ad2d6701b04e0" exitCode=60
Nov 03 22:15:03.100927 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.100495    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" event={"ID":"dee72c2b-3938-457a-98a3-565450f0c7ed","Type":"ContainerDied","Data":"06ffae7bd575b7c0116765d1d3b67f08c08214ef9ee97f18807ad2d6701b04e0"}
Nov 03 22:15:03.100927 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.100567    5244 scope.go:122] "RemoveContainer" containerID="971f311bde8e526454c76bb31e449bdfa61015d0bea2d98ec544907c475f71db"
Nov 03 22:15:03.100927 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.100841    5244 scope.go:122] "RemoveContainer" containerID="06ffae7bd575b7c0116765d1d3b67f08c08214ef9ee97f18807ad2d6701b04e0"
Nov 03 22:15:03.101171 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:03.101151    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2_odh-ai-gateway-infra(dee72c2b-3938-457a-98a3-565450f0c7ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed"
Nov 03 22:15:03.102177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.102157    5244 generic.go:356] "Generic (PLEG): container finished" podID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerID="0ba0b8cccf9c023e5117be9a637f82de35213c33cd8efe004f2737935d51ffcf" exitCode=60
Nov 03 22:15:03.102268 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.102241    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerDied","Data":"0ba0b8cccf9c023e5117be9a637f82de35213c33cd8efe004f2737935d51ffcf"}
Nov 03 22:15:03.102559 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.102541    5244 scope.go:122] "RemoveContainer" containerID="0ba0b8cccf9c023e5117be9a637f82de35213c33cd8efe004f2737935d51ffcf"
Nov 03 22:15:03.104008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.103980    5244 generic.go:356] "Generic (PLEG): container finished" podID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerID="3dd57394508274b54fc3c951823977abadc787da7dfa2e090355cc9103107137" exitCode=60
Nov 03 22:15:03.104137 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.104067    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerDied","Data":"3dd57394508274b54fc3c951823977abadc787da7dfa2e090355cc9103107137"}
Nov 03 22:15:03.104298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.104283    5244 scope.go:122] "RemoveContainer" containerID="3dd57394508274b54fc3c951823977abadc787da7dfa2e090355cc9103107137"
Nov 03 22:15:03.105965 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.105940    5244 generic.go:356] "Generic (PLEG): container finished" podID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerID="f51f35ddda0f734fd1504d9f116419d05d55cb6d329295fbafc0490cd152f36f" exitCode=60
Nov 03 22:15:03.106102 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.106024    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerDied","Data":"f51f35ddda0f734fd1504d9f116419d05d55cb6d329295fbafc0490cd152f36f"}
Nov 03 22:15:03.106569 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.106549    5244 scope.go:122] "RemoveContainer" containerID="f51f35ddda0f734fd1504d9f116419d05d55cb6d329295fbafc0490cd152f36f"
Nov 03 22:15:03.108842 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.108822    5244 generic.go:356] "Generic (PLEG): container finished" podID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerID="6471237c99cad1a8db7222149686ef77f8d8fed21390a5102b082d606f092d20" exitCode=60
Nov 03 22:15:03.108917 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.108891    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerDied","Data":"6471237c99cad1a8db7222149686ef77f8d8fed21390a5102b082d606f092d20"}
Nov 03 22:15:03.109159 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.109141    5244 scope.go:122] "RemoveContainer" containerID="6471237c99cad1a8db7222149686ef77f8d8fed21390a5102b082d606f092d20"
Nov 03 22:15:03.110713 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.110690    5244 generic.go:356] "Generic (PLEG): container finished" podID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerID="97e9eda8f9bc19825f80ba83467cf5cfb8738932cbefd9abd35dc09402595b44" exitCode=60
Nov 03 22:15:03.110797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.110778    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerDied","Data":"97e9eda8f9bc19825f80ba83467cf5cfb8738932cbefd9abd35dc09402595b44"}
Nov 03 22:15:03.110985 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.110967    5244 scope.go:122] "RemoveContainer" containerID="97e9eda8f9bc19825f80ba83467cf5cfb8738932cbefd9abd35dc09402595b44"
Nov 03 22:15:03.113120 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.113097    5244 generic.go:356] "Generic (PLEG): container finished" podID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerID="03b8007e3dc13ef2476bb3fac989439fd933edf12a00bc6b4bed31ce89f81053" exitCode=60
Nov 03 22:15:03.113237 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.113174    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerDied","Data":"03b8007e3dc13ef2476bb3fac989439fd933edf12a00bc6b4bed31ce89f81053"}
Nov 03 22:15:03.113404 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.113387    5244 scope.go:122] "RemoveContainer" containerID="03b8007e3dc13ef2476bb3fac989439fd933edf12a00bc6b4bed31ce89f81053"
Nov 03 22:15:03.115575 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.115526    5244 generic.go:356] "Generic (PLEG): container finished" podID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerID="8685b7aeedc151164eb9acbeb7664b35cc9eb718968298835ad5cd1537306c79" exitCode=60
Nov 03 22:15:03.115673 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.115574    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" event={"ID":"499fbb98-6d07-4117-8dd1-99178f6fbcc3","Type":"ContainerDied","Data":"8685b7aeedc151164eb9acbeb7664b35cc9eb718968298835ad5cd1537306c79"}
Nov 03 22:15:03.115783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.115769    5244 scope.go:122] "RemoveContainer" containerID="8685b7aeedc151164eb9acbeb7664b35cc9eb718968298835ad5cd1537306c79"
Nov 03 22:15:03.115952 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:03.115931    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j_odh-ai-gateway-infra(499fbb98-6d07-4117-8dd1-99178f6fbcc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3"
Nov 03 22:15:03.179936 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:03.179909    5244 scope.go:122] "RemoveContainer" containerID="e22f2a0bb15b947c5318dbf4a56b8ac9ecfe61aa5e6723f609cb5cf1acab9a86"
Nov 03 22:15:04.126882 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.126844    5244 scope.go:122] "RemoveContainer" containerID="06ffae7bd575b7c0116765d1d3b67f08c08214ef9ee97f18807ad2d6701b04e0"
Nov 03 22:15:04.127405 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:04.127150    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2_odh-ai-gateway-infra(dee72c2b-3938-457a-98a3-565450f0c7ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed"
Nov 03 22:15:04.128628 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.128601    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerStarted","Data":"f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3"}
Nov 03 22:15:04.130672 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.130643    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerStarted","Data":"01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7"}
Nov 03 22:15:04.136412 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.135600    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerStarted","Data":"298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f"}
Nov 03 22:15:04.138911 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.138877    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerStarted","Data":"e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab"}
Nov 03 22:15:04.143018 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.142951    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerStarted","Data":"c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304"}
Nov 03 22:15:04.145659 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.145628    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerStarted","Data":"44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df"}
Nov 03 22:15:04.147646 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:04.147623    5244 scope.go:122] "RemoveContainer" containerID="8685b7aeedc151164eb9acbeb7664b35cc9eb718968298835ad5cd1537306c79"
Nov 03 22:15:04.147877 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:04.147857    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j_odh-ai-gateway-infra(499fbb98-6d07-4117-8dd1-99178f6fbcc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3"
Nov 03 22:15:05.155949 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.155914    5244 generic.go:356] "Generic (PLEG): container finished" podID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerID="01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7" exitCode=60
Nov 03 22:15:05.156442 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.155992    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerDied","Data":"01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7"}
Nov 03 22:15:05.156442 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.156053    5244 scope.go:122] "RemoveContainer" containerID="3dd57394508274b54fc3c951823977abadc787da7dfa2e090355cc9103107137"
Nov 03 22:15:05.156442 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.156257    5244 scope.go:122] "RemoveContainer" containerID="01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7"
Nov 03 22:15:05.156618 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:05.156602    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv_odh-ai-gateway-infra(ebbe2af4-e992-497b-a95a-d672a29c842e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e"
Nov 03 22:15:05.158057 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.158035    5244 generic.go:356] "Generic (PLEG): container finished" podID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerID="298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f" exitCode=60
Nov 03 22:15:05.158169 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.158107    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerDied","Data":"298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f"}
Nov 03 22:15:05.158431 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.158413    5244 scope.go:122] "RemoveContainer" containerID="298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f"
Nov 03 22:15:05.158711 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:05.158687    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj_odh-ai-gateway-infra(50da3769-6de9-4b39-b217-3da2b24d98ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff"
Nov 03 22:15:05.160131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.160106    5244 generic.go:356] "Generic (PLEG): container finished" podID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerID="e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab" exitCode=60
Nov 03 22:15:05.160251 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.160200    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerDied","Data":"e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab"}
Nov 03 22:15:05.160424 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.160409    5244 scope.go:122] "RemoveContainer" containerID="e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab"
Nov 03 22:15:05.160690 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:05.160667    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn_odh-ai-gateway-infra(6c6fb05c-594a-4e9a-bf73-e241622d52e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0"
Nov 03 22:15:05.162208 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.162188    5244 generic.go:356] "Generic (PLEG): container finished" podID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerID="c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304" exitCode=60
Nov 03 22:15:05.162305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.162284    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerDied","Data":"c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304"}
Nov 03 22:15:05.162626 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.162607    5244 scope.go:122] "RemoveContainer" containerID="c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304"
Nov 03 22:15:05.162915 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:05.162890    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8_odh-ai-gateway-infra(1a77b83c-515e-415a-b375-0c72b0abfb85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85"
Nov 03 22:15:05.164462 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.164439    5244 generic.go:356] "Generic (PLEG): container finished" podID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerID="44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df" exitCode=60
Nov 03 22:15:05.164768 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.164742    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerDied","Data":"44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df"}
Nov 03 22:15:05.164902 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.164829    5244 scope.go:122] "RemoveContainer" containerID="44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df"
Nov 03 22:15:05.165071 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:05.165049    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9_odh-ai-gateway-infra(f09a28fe-6665-44c3-90fa-c3dc9035b301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301"
Nov 03 22:15:05.166692 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.166670    5244 generic.go:356] "Generic (PLEG): container finished" podID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerID="f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3" exitCode=60
Nov 03 22:15:05.166808 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.166718    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerDied","Data":"f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3"}
Nov 03 22:15:05.166958 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.166941    5244 scope.go:122] "RemoveContainer" containerID="f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3"
Nov 03 22:15:05.167180 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:05.167161    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d_odh-ai-gateway-infra(2652d3bb-1745-4a5e-8921-c84792ec9f34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34"
Nov 03 22:15:05.169806 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.169790    5244 scope.go:122] "RemoveContainer" containerID="f51f35ddda0f734fd1504d9f116419d05d55cb6d329295fbafc0490cd152f36f"
Nov 03 22:15:05.176261 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.176241    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:15:05.183253 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.183236    5244 scope.go:122] "RemoveContainer" containerID="6471237c99cad1a8db7222149686ef77f8d8fed21390a5102b082d606f092d20"
Nov 03 22:15:05.196067 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.196043    5244 scope.go:122] "RemoveContainer" containerID="97e9eda8f9bc19825f80ba83467cf5cfb8738932cbefd9abd35dc09402595b44"
Nov 03 22:15:05.209051 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.209027    5244 scope.go:122] "RemoveContainer" containerID="03b8007e3dc13ef2476bb3fac989439fd933edf12a00bc6b4bed31ce89f81053"
Nov 03 22:15:05.221750 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:05.221728    5244 scope.go:122] "RemoveContainer" containerID="0ba0b8cccf9c023e5117be9a637f82de35213c33cd8efe004f2737935d51ffcf"
Nov 03 22:15:06.177195 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:06.177154    5244 scope.go:122] "RemoveContainer" containerID="44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df"
Nov 03 22:15:06.177657 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:06.177468    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9_odh-ai-gateway-infra(f09a28fe-6665-44c3-90fa-c3dc9035b301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301"
Nov 03 22:15:06.178965 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:06.178943    5244 scope.go:122] "RemoveContainer" containerID="f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3"
Nov 03 22:15:06.179204 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:06.179183    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d_odh-ai-gateway-infra(2652d3bb-1745-4a5e-8921-c84792ec9f34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34"
Nov 03 22:15:06.180596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:06.180577    5244 scope.go:122] "RemoveContainer" containerID="01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7"
Nov 03 22:15:06.180821 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:06.180802    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv_odh-ai-gateway-infra(ebbe2af4-e992-497b-a95a-d672a29c842e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e"
Nov 03 22:15:06.182183 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:06.182158    5244 scope.go:122] "RemoveContainer" containerID="298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f"
Nov 03 22:15:06.182415 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:06.182392    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj_odh-ai-gateway-infra(50da3769-6de9-4b39-b217-3da2b24d98ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff"
Nov 03 22:15:06.183699 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:06.183680    5244 scope.go:122] "RemoveContainer" containerID="e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab"
Nov 03 22:15:06.183884 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:06.183863    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn_odh-ai-gateway-infra(6c6fb05c-594a-4e9a-bf73-e241622d52e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0"
Nov 03 22:15:06.185289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:06.185274    5244 scope.go:122] "RemoveContainer" containerID="c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304"
Nov 03 22:15:06.185481 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:06.185464    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8_odh-ai-gateway-infra(1a77b83c-515e-415a-b375-0c72b0abfb85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85"
Nov 03 22:15:12.099148 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:12.099114    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:15:12.099740 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:12.099717    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:15:13.052275 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:13.052232    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"]
Nov 03 22:15:13.100309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:13.100275    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6175ce20-2tb9t"]
Nov 03 22:15:13.136739 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:13.136707    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ec4513-7ac4-4147-9a77-189d818b063d" path="/var/lib/kubelet/pods/f6ec4513-7ac4-4147-9a77-189d818b063d/volumes"
Nov 03 22:15:14.129524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:14.129491    5244 scope.go:122] "RemoveContainer" containerID="06ffae7bd575b7c0116765d1d3b67f08c08214ef9ee97f18807ad2d6701b04e0"
Nov 03 22:15:15.293430 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:15.293345    5244 generic.go:356] "Generic (PLEG): container finished" podID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerID="53a18d05ce916f987a86928a6a060bcdac566d7edaa7a3219dd77101040f3965" exitCode=60
Nov 03 22:15:15.293430 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:15.293376    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" event={"ID":"dee72c2b-3938-457a-98a3-565450f0c7ed","Type":"ContainerDied","Data":"53a18d05ce916f987a86928a6a060bcdac566d7edaa7a3219dd77101040f3965"}
Nov 03 22:15:15.294027 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:15.293456    5244 scope.go:122] "RemoveContainer" containerID="06ffae7bd575b7c0116765d1d3b67f08c08214ef9ee97f18807ad2d6701b04e0"
Nov 03 22:15:15.294027 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:15.293827    5244 scope.go:122] "RemoveContainer" containerID="53a18d05ce916f987a86928a6a060bcdac566d7edaa7a3219dd77101040f3965"
Nov 03 22:15:15.294140 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:15.294120    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2_odh-ai-gateway-infra(dee72c2b-3938-457a-98a3-565450f0c7ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed"
Nov 03 22:15:16.130290 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.130255    5244 scope.go:122] "RemoveContainer" containerID="f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3"
Nov 03 22:15:16.130516 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.130312    5244 scope.go:122] "RemoveContainer" containerID="01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7"
Nov 03 22:15:16.130516 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.130418    5244 scope.go:122] "RemoveContainer" containerID="c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304"
Nov 03 22:15:16.416814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.416727    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"]
Nov 03 22:15:16.579006 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.578984    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:16.638579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.638483    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca\") pod \"dee72c2b-3938-457a-98a3-565450f0c7ed\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") "
Nov 03 22:15:16.638579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.638521    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb\" (UniqueName: \"kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb\") pod \"dee72c2b-3938-457a-98a3-565450f0c7ed\" (UID: \"dee72c2b-3938-457a-98a3-565450f0c7ed\") "
Nov 03 22:15:16.638896 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.638861    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca" pod "dee72c2b-3938-457a-98a3-565450f0c7ed" (UID: "dee72c2b-3938-457a-98a3-565450f0c7ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:16.639070 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.638999    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee72c2b-3938-457a-98a3-565450f0c7ed-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:16.640864 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.640825    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb" pod "dee72c2b-3938-457a-98a3-565450f0c7ed" (UID: "dee72c2b-3938-457a-98a3-565450f0c7ed"). InnerVolumeSpecName "kube-api-access-t56fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:16.740041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:16.740002    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56fb\" (UniqueName: \"kubernetes.io/projected/dee72c2b-3938-457a-98a3-565450f0c7ed-kube-api-access-t56fb\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:17.130237 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.130176    5244 scope.go:122] "RemoveContainer" containerID="e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab"
Nov 03 22:15:17.130473 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.130435    5244 scope.go:122] "RemoveContainer" containerID="8685b7aeedc151164eb9acbeb7664b35cc9eb718968298835ad5cd1537306c79"
Nov 03 22:15:17.310613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.310524    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"
Nov 03 22:15:17.310613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.310519    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2" event={"ID":"dee72c2b-3938-457a-98a3-565450f0c7ed","Type":"ContainerDied","Data":"918f1a97147ea9e172f2a5c62e01b22733c1454c0e9275231a6aa20ceefcedfb"}
Nov 03 22:15:17.310856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.310675    5244 scope.go:122] "RemoveContainer" containerID="53a18d05ce916f987a86928a6a060bcdac566d7edaa7a3219dd77101040f3965"
Nov 03 22:15:17.313276 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.313241    5244 generic.go:356] "Generic (PLEG): container finished" podID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerID="6d399b3536bef91b8d1f930e67ba9813523fb1caa20109d39b9e06b1fb401432" exitCode=60
Nov 03 22:15:17.313440 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.313318    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerDied","Data":"6d399b3536bef91b8d1f930e67ba9813523fb1caa20109d39b9e06b1fb401432"}
Nov 03 22:15:17.313950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.313926    5244 scope.go:122] "RemoveContainer" containerID="6d399b3536bef91b8d1f930e67ba9813523fb1caa20109d39b9e06b1fb401432"
Nov 03 22:15:17.314288 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:17.314261    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d_odh-ai-gateway-infra(2652d3bb-1745-4a5e-8921-c84792ec9f34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34"
Nov 03 22:15:17.316130 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.316108    5244 generic.go:356] "Generic (PLEG): container finished" podID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerID="5dde5276bc0ba4a57cb35c878183467d253d66354aef83b0eb95c1150b2602ef" exitCode=60
Nov 03 22:15:17.316234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.316178    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerDied","Data":"5dde5276bc0ba4a57cb35c878183467d253d66354aef83b0eb95c1150b2602ef"}
Nov 03 22:15:17.316579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.316561    5244 scope.go:122] "RemoveContainer" containerID="5dde5276bc0ba4a57cb35c878183467d253d66354aef83b0eb95c1150b2602ef"
Nov 03 22:15:17.316848 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:17.316823    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv_odh-ai-gateway-infra(ebbe2af4-e992-497b-a95a-d672a29c842e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e"
Nov 03 22:15:17.318292 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.318272    5244 generic.go:356] "Generic (PLEG): container finished" podID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerID="52b2b20a6293bbe0c6a22ebc9e06d414be2456feaebe22bda8a6f45a614ddf3a" exitCode=60
Nov 03 22:15:17.318389 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.318353    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerDied","Data":"52b2b20a6293bbe0c6a22ebc9e06d414be2456feaebe22bda8a6f45a614ddf3a"}
Nov 03 22:15:17.318646 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.318632    5244 scope.go:122] "RemoveContainer" containerID="52b2b20a6293bbe0c6a22ebc9e06d414be2456feaebe22bda8a6f45a614ddf3a"
Nov 03 22:15:17.318851 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:17.318834    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8_odh-ai-gateway-infra(1a77b83c-515e-415a-b375-0c72b0abfb85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85"
Nov 03 22:15:17.324149 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.324129    5244 scope.go:122] "RemoveContainer" containerID="f2dbb2a16d9ca5f5c2a6e84fab4ec14b632829240d76293cfac1c90aa73610e3"
Nov 03 22:15:17.339856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.339835    5244 scope.go:122] "RemoveContainer" containerID="01708ba43aa9d0860cf584cf20db2ff3297944dff9a9dd821ca4087571fe11c7"
Nov 03 22:15:17.353552 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.353529    5244 scope.go:122] "RemoveContainer" containerID="c83d05be95540cad98ca170e54289b7bec4403ce46153f55ff24ec843ff3d304"
Nov 03 22:15:17.585313 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.585219    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"]
Nov 03 22:15:17.588831 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:17.588806    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a49a32-29895735-2tnk2"]
Nov 03 22:15:18.129900 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.129867    5244 scope.go:122] "RemoveContainer" containerID="44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df"
Nov 03 22:15:18.130187 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.130170    5244 scope.go:122] "RemoveContainer" containerID="298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f"
Nov 03 22:15:18.332456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.332374    5244 generic.go:356] "Generic (PLEG): container finished" podID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerID="c4bc37260e037e53f56c71bfa4390fca17196e19e25c1bf9bbf1b250309bdf8b" exitCode=60
Nov 03 22:15:18.332456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.332439    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerDied","Data":"c4bc37260e037e53f56c71bfa4390fca17196e19e25c1bf9bbf1b250309bdf8b"}
Nov 03 22:15:18.332658 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.332501    5244 scope.go:122] "RemoveContainer" containerID="e2412396f17e8cce469aa9dd0a80092acf752f356a9ab7d96a8a491ccb6f3cab"
Nov 03 22:15:18.332924 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.332899    5244 scope.go:122] "RemoveContainer" containerID="c4bc37260e037e53f56c71bfa4390fca17196e19e25c1bf9bbf1b250309bdf8b"
Nov 03 22:15:18.333279 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:18.333251    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn_odh-ai-gateway-infra(6c6fb05c-594a-4e9a-bf73-e241622d52e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0"
Nov 03 22:15:18.335851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.335831    5244 generic.go:356] "Generic (PLEG): container finished" podID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerID="85725c2a2a72f1f070b6bb2eb47463eaa117548cfa5a22c0ac283dbc2896c205" exitCode=60
Nov 03 22:15:18.335937 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.335907    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" event={"ID":"499fbb98-6d07-4117-8dd1-99178f6fbcc3","Type":"ContainerDied","Data":"85725c2a2a72f1f070b6bb2eb47463eaa117548cfa5a22c0ac283dbc2896c205"}
Nov 03 22:15:18.336316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.336249    5244 scope.go:122] "RemoveContainer" containerID="85725c2a2a72f1f070b6bb2eb47463eaa117548cfa5a22c0ac283dbc2896c205"
Nov 03 22:15:18.336523 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:18.336500    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j_odh-ai-gateway-infra(499fbb98-6d07-4117-8dd1-99178f6fbcc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3"
Nov 03 22:15:18.345866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.345842    5244 scope.go:122] "RemoveContainer" containerID="8685b7aeedc151164eb9acbeb7664b35cc9eb718968298835ad5cd1537306c79"
Nov 03 22:15:18.485656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.485614    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"]
Nov 03 22:15:18.663022 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.662989    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:18.708689 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.708658    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"]
Nov 03 22:15:18.763215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.763183    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca\") pod \"1a77b83c-515e-415a-b375-0c72b0abfb85\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") "
Nov 03 22:15:18.763439 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.763247    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck\" (UniqueName: \"kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck\") pod \"1a77b83c-515e-415a-b375-0c72b0abfb85\" (UID: \"1a77b83c-515e-415a-b375-0c72b0abfb85\") "
Nov 03 22:15:18.763659 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.763634    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca" pod "1a77b83c-515e-415a-b375-0c72b0abfb85" (UID: "1a77b83c-515e-415a-b375-0c72b0abfb85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:18.765585 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.765563    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck" pod "1a77b83c-515e-415a-b375-0c72b0abfb85" (UID: "1a77b83c-515e-415a-b375-0c72b0abfb85"). InnerVolumeSpecName "kube-api-access-tbrck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:18.827572 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.827538    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"]
Nov 03 22:15:18.848588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.848560    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:18.865541 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.864447    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a77b83c-515e-415a-b375-0c72b0abfb85-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.865541 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.864481    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbrck\" (UniqueName: \"kubernetes.io/projected/1a77b83c-515e-415a-b375-0c72b0abfb85-kube-api-access-tbrck\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.965290 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.965264    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx\" (UniqueName: \"kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx\") pod \"2652d3bb-1745-4a5e-8921-c84792ec9f34\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") "
Nov 03 22:15:18.965490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.965318    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca\") pod \"2652d3bb-1745-4a5e-8921-c84792ec9f34\" (UID: \"2652d3bb-1745-4a5e-8921-c84792ec9f34\") "
Nov 03 22:15:18.965684 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.965660    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca" pod "2652d3bb-1745-4a5e-8921-c84792ec9f34" (UID: "2652d3bb-1745-4a5e-8921-c84792ec9f34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:18.965838 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.965819    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2652d3bb-1745-4a5e-8921-c84792ec9f34-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:18.967818 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.967796    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx" pod "2652d3bb-1745-4a5e-8921-c84792ec9f34" (UID: "2652d3bb-1745-4a5e-8921-c84792ec9f34"). InnerVolumeSpecName "kube-api-access-kwbgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:18.969502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:18.969482    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:19.066287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.066247    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca\") pod \"ebbe2af4-e992-497b-a95a-d672a29c842e\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") "
Nov 03 22:15:19.066478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.066420    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw\" (UniqueName: \"kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw\") pod \"ebbe2af4-e992-497b-a95a-d672a29c842e\" (UID: \"ebbe2af4-e992-497b-a95a-d672a29c842e\") "
Nov 03 22:15:19.066697 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.066667    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca" pod "ebbe2af4-e992-497b-a95a-d672a29c842e" (UID: "ebbe2af4-e992-497b-a95a-d672a29c842e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:19.066779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.066708    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwbgx\" (UniqueName: \"kubernetes.io/projected/2652d3bb-1745-4a5e-8921-c84792ec9f34-kube-api-access-kwbgx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:19.068609 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.068582    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw" pod "ebbe2af4-e992-497b-a95a-d672a29c842e" (UID: "ebbe2af4-e992-497b-a95a-d672a29c842e"). InnerVolumeSpecName "kube-api-access-wk7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:19.138490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.138461    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" path="/var/lib/kubelet/pods/dee72c2b-3938-457a-98a3-565450f0c7ed/volumes"
Nov 03 22:15:19.167638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.167582    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7dw\" (UniqueName: \"kubernetes.io/projected/ebbe2af4-e992-497b-a95a-d672a29c842e-kube-api-access-wk7dw\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:19.167638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.167605    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbe2af4-e992-497b-a95a-d672a29c842e-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:19.346890 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.346861    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"
Nov 03 22:15:19.346890 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.346875    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv" event={"ID":"ebbe2af4-e992-497b-a95a-d672a29c842e","Type":"ContainerDied","Data":"22dbd0655d9d730ffd6a551e3826a007b274b263b3043cd89d0a067f03cb53e1"}
Nov 03 22:15:19.347141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.346952    5244 scope.go:122] "RemoveContainer" containerID="5dde5276bc0ba4a57cb35c878183467d253d66354aef83b0eb95c1150b2602ef"
Nov 03 22:15:19.349238 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.349116    5244 generic.go:356] "Generic (PLEG): container finished" podID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerID="92c4704c37d7ac7470ab3ba21940eb8535302bc756275abfe4629be525e4e439" exitCode=60
Nov 03 22:15:19.349238 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.349209    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerDied","Data":"92c4704c37d7ac7470ab3ba21940eb8535302bc756275abfe4629be525e4e439"}
Nov 03 22:15:19.349712 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.349694    5244 scope.go:122] "RemoveContainer" containerID="92c4704c37d7ac7470ab3ba21940eb8535302bc756275abfe4629be525e4e439"
Nov 03 22:15:19.350013 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:19.349986    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj_odh-ai-gateway-infra(50da3769-6de9-4b39-b217-3da2b24d98ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff"
Nov 03 22:15:19.352748 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.352730    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"
Nov 03 22:15:19.352855 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.352765    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8" event={"ID":"1a77b83c-515e-415a-b375-0c72b0abfb85","Type":"ContainerDied","Data":"993efcf917fa1058502e709a9a4594d76f8561b388a2101a457b0da4d11a4fc9"}
Nov 03 22:15:19.354988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.354963    5244 generic.go:356] "Generic (PLEG): container finished" podID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerID="8930bba5879c58ecc3b9aeca611bc8960c6e38a5e3018e32ab072558885acc94" exitCode=60
Nov 03 22:15:19.355104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.355035    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerDied","Data":"8930bba5879c58ecc3b9aeca611bc8960c6e38a5e3018e32ab072558885acc94"}
Nov 03 22:15:19.355452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.355431    5244 scope.go:122] "RemoveContainer" containerID="8930bba5879c58ecc3b9aeca611bc8960c6e38a5e3018e32ab072558885acc94"
Nov 03 22:15:19.355709 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:19.355688    5244 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9_odh-ai-gateway-infra(f09a28fe-6665-44c3-90fa-c3dc9035b301)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301"
Nov 03 22:15:19.358103 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.358084    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d" event={"ID":"2652d3bb-1745-4a5e-8921-c84792ec9f34","Type":"ContainerDied","Data":"856a6aede35265b5d6f3db22ec5a641f21b7d1352e87b1f1510c6e075bc796be"}
Nov 03 22:15:19.358213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.358106    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"
Nov 03 22:15:19.358560 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.358542    5244 scope.go:122] "RemoveContainer" containerID="298e62d5375b0280088b642c47248472d2e302093b99c291552c10629c509c2f"
Nov 03 22:15:19.372872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.372849    5244 scope.go:122] "RemoveContainer" containerID="52b2b20a6293bbe0c6a22ebc9e06d414be2456feaebe22bda8a6f45a614ddf3a"
Nov 03 22:15:19.381637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.381617    5244 scope.go:122] "RemoveContainer" containerID="44922f3c614392ac9b99a92826c2394607c335fd8bd5bbd2343978cf0f14a1df"
Nov 03 22:15:19.394355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.394303    5244 scope.go:122] "RemoveContainer" containerID="6d399b3536bef91b8d1f930e67ba9813523fb1caa20109d39b9e06b1fb401432"
Nov 03 22:15:19.459141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.459112    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"]
Nov 03 22:15:19.512376 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.512340    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"]
Nov 03 22:15:19.535709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.530982    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-87e1cf-29895735-cwtx8"]
Nov 03 22:15:19.572865 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.572833    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"]
Nov 03 22:15:19.623589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.623553    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:19.632761 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.632731    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"]
Nov 03 22:15:19.643183 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.642489    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-da5f0b-29895735-dcd2d"]
Nov 03 22:15:19.738004 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.737974    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:19.776578 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.776535    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca\") pod \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") "
Nov 03 22:15:19.776785 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.776686    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq\" (UniqueName: \"kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq\") pod \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\" (UID: \"6c6fb05c-594a-4e9a-bf73-e241622d52e0\") "
Nov 03 22:15:19.777008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.776980    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca" pod "6c6fb05c-594a-4e9a-bf73-e241622d52e0" (UID: "6c6fb05c-594a-4e9a-bf73-e241622d52e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:19.777188 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.777162    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fb05c-594a-4e9a-bf73-e241622d52e0-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:19.778865 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.778844    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq" pod "6c6fb05c-594a-4e9a-bf73-e241622d52e0" (UID: "6c6fb05c-594a-4e9a-bf73-e241622d52e0"). InnerVolumeSpecName "kube-api-access-9mlzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:19.861452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.861419    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"]
Nov 03 22:15:19.863590 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.863566    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e15bbc-29895735-z2hgv"]
Nov 03 22:15:19.878180 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.878148    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj\" (UniqueName: \"kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj\") pod \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") "
Nov 03 22:15:19.878347 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.878189    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca\") pod \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\" (UID: \"499fbb98-6d07-4117-8dd1-99178f6fbcc3\") "
Nov 03 22:15:19.878660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.878640    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca" pod "499fbb98-6d07-4117-8dd1-99178f6fbcc3" (UID: "499fbb98-6d07-4117-8dd1-99178f6fbcc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:19.878660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.878652    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlzq\" (UniqueName: \"kubernetes.io/projected/6c6fb05c-594a-4e9a-bf73-e241622d52e0-kube-api-access-9mlzq\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:19.880390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.880369    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj" pod "499fbb98-6d07-4117-8dd1-99178f6fbcc3" (UID: "499fbb98-6d07-4117-8dd1-99178f6fbcc3"). InnerVolumeSpecName "kube-api-access-bskbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:19.979613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.979525    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskbj\" (UniqueName: \"kubernetes.io/projected/499fbb98-6d07-4117-8dd1-99178f6fbcc3-kube-api-access-bskbj\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:19.979613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:19.979551    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/499fbb98-6d07-4117-8dd1-99178f6fbcc3-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:20.367726 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.367690    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"
Nov 03 22:15:20.367726 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.367715    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j" event={"ID":"499fbb98-6d07-4117-8dd1-99178f6fbcc3","Type":"ContainerDied","Data":"bdcc6267bd2707ce8e73ea0a7b2ccbaec95cb3c0de6ca0d83b1236e222a95a3d"}
Nov 03 22:15:20.368012 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.367786    5244 scope.go:122] "RemoveContainer" containerID="85725c2a2a72f1f070b6bb2eb47463eaa117548cfa5a22c0ac283dbc2896c205"
Nov 03 22:15:20.373721 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.373698    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"
Nov 03 22:15:20.373861 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.373698    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn" event={"ID":"6c6fb05c-594a-4e9a-bf73-e241622d52e0","Type":"ContainerDied","Data":"91c745f5f6694a5ddb9aba66695c2da229b88436329d25124790a3b8a2a838ed"}
Nov 03 22:15:20.380305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.380283    5244 scope.go:122] "RemoveContainer" containerID="c4bc37260e037e53f56c71bfa4390fca17196e19e25c1bf9bbf1b250309bdf8b"
Nov 03 22:15:20.532840 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.532750    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"]
Nov 03 22:15:20.535939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.535907    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-0d45ba-29895735-ctc9j"]
Nov 03 22:15:20.643468 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.643433    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"]
Nov 03 22:15:20.647240 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.647212    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6de8e6-29895735-rgrqn"]
Nov 03 22:15:20.761151 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.761101    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"]
Nov 03 22:15:20.919428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.919405    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:20.981344 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.981303    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"]
Nov 03 22:15:20.992853 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.991289    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca\") pod \"50da3769-6de9-4b39-b217-3da2b24d98ff\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") "
Nov 03 22:15:20.992853 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.991378    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n\" (UniqueName: \"kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n\") pod \"50da3769-6de9-4b39-b217-3da2b24d98ff\" (UID: \"50da3769-6de9-4b39-b217-3da2b24d98ff\") "
Nov 03 22:15:21.010629 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:20.998789    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca" pod "50da3769-6de9-4b39-b217-3da2b24d98ff" (UID: "50da3769-6de9-4b39-b217-3da2b24d98ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:21.078071 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.077540    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n" pod "50da3769-6de9-4b39-b217-3da2b24d98ff" (UID: "50da3769-6de9-4b39-b217-3da2b24d98ff"). InnerVolumeSpecName "kube-api-access-4n69n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:21.093423 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.093389    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50da3769-6de9-4b39-b217-3da2b24d98ff-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:21.093423 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.093424    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n69n\" (UniqueName: \"kubernetes.io/projected/50da3769-6de9-4b39-b217-3da2b24d98ff-kube-api-access-4n69n\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:21.141104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.141072    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" path="/var/lib/kubelet/pods/1a77b83c-515e-415a-b375-0c72b0abfb85/volumes"
Nov 03 22:15:21.141682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.141663    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" path="/var/lib/kubelet/pods/2652d3bb-1745-4a5e-8921-c84792ec9f34/volumes"
Nov 03 22:15:21.142193 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.142167    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" path="/var/lib/kubelet/pods/499fbb98-6d07-4117-8dd1-99178f6fbcc3/volumes"
Nov 03 22:15:21.143128 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.143108    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" path="/var/lib/kubelet/pods/6c6fb05c-594a-4e9a-bf73-e241622d52e0/volumes"
Nov 03 22:15:21.143703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.143679    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" path="/var/lib/kubelet/pods/ebbe2af4-e992-497b-a95a-d672a29c842e/volumes"
Nov 03 22:15:21.231755 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.231725    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:21.307339 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.307300    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj\" (UniqueName: \"kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj\") pod \"f09a28fe-6665-44c3-90fa-c3dc9035b301\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") "
Nov 03 22:15:21.307492 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.307473    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca\") pod \"f09a28fe-6665-44c3-90fa-c3dc9035b301\" (UID: \"f09a28fe-6665-44c3-90fa-c3dc9035b301\") "
Nov 03 22:15:21.307851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.307819    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca" pod "f09a28fe-6665-44c3-90fa-c3dc9035b301" (UID: "f09a28fe-6665-44c3-90fa-c3dc9035b301"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:15:21.309635 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.309609    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj" pod "f09a28fe-6665-44c3-90fa-c3dc9035b301" (UID: "f09a28fe-6665-44c3-90fa-c3dc9035b301"). InnerVolumeSpecName "kube-api-access-kwkvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:15:21.407963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.403199    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj" event={"ID":"50da3769-6de9-4b39-b217-3da2b24d98ff","Type":"ContainerDied","Data":"1f83c1d072feb03deed0ff9c951f772fd25ed77816b31043d4aa7eb5379421af"}
Nov 03 22:15:21.407963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.403275    5244 scope.go:122] "RemoveContainer" containerID="92c4704c37d7ac7470ab3ba21940eb8535302bc756275abfe4629be525e4e439"
Nov 03 22:15:21.407963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.403423    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"
Nov 03 22:15:21.423452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.409520    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkvj\" (UniqueName: \"kubernetes.io/projected/f09a28fe-6665-44c3-90fa-c3dc9035b301-kube-api-access-kwkvj\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:21.423452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.409553    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f09a28fe-6665-44c3-90fa-c3dc9035b301-maas-api-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:15:21.423452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.416079    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9" event={"ID":"f09a28fe-6665-44c3-90fa-c3dc9035b301","Type":"ContainerDied","Data":"9afb02f42cf2c8783664a445ce7903edad59de7c2dc32f22fcfaa4535913b438"}
Nov 03 22:15:21.423452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.416204    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"
Nov 03 22:15:21.444719 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.444676    5244 scope.go:122] "RemoveContainer" containerID="8930bba5879c58ecc3b9aeca611bc8960c6e38a5e3018e32ab072558885acc94"
Nov 03 22:15:21.567854 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.567789    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"]
Nov 03 22:15:21.570181 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.570111    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7ee809-29895735-vk7wj"]
Nov 03 22:15:21.677670 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.677633    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"]
Nov 03 22:15:21.688452 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:21.683801    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5b1489-29895735-4pzg9"]
Nov 03 22:15:22.439542 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:22.439502    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:15:23.044524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:23.044489    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"]
Nov 03 22:15:23.093586 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:23.093551    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e35cc-c88hp"]
Nov 03 22:15:23.137454 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:23.137424    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257d40d9-7d49-4ab0-bf3c-15626215e08a" path="/var/lib/kubelet/pods/257d40d9-7d49-4ab0-bf3c-15626215e08a/volumes"
Nov 03 22:15:23.137788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:23.137775    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" path="/var/lib/kubelet/pods/50da3769-6de9-4b39-b217-3da2b24d98ff/volumes"
Nov 03 22:15:23.138159 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:23.138145    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" path="/var/lib/kubelet/pods/f09a28fe-6665-44c3-90fa-c3dc9035b301/volumes"
Nov 03 22:15:30.515789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:30.515754    5244 scope.go:122] "RemoveContainer" containerID="d922bd9756d81cecbd00dcff59955a8ab924f8befdb59e084d29f21e6d537e5c"
Nov 03 22:15:34.442199 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442154    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"]
Nov 03 22:15:34.442843 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442823    5244 cpu_manager.go:414] "containerMap: removing container" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.442894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442848    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.442894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442869    5244 cpu_manager.go:414] "containerMap: removing container" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:15:34.442894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442877    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:15:34.442894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442888    5244 cpu_manager.go:414] "containerMap: removing container" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442895    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442911    5244 cpu_manager.go:414] "containerMap: removing container" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442917    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442926    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442933    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442943    5244 cpu_manager.go:414] "containerMap: removing container" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442949    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442957    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442963    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442975    5244 cpu_manager.go:414] "containerMap: removing container" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442982    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442991    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.442997    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443010    5244 cpu_manager.go:414] "containerMap: removing container" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443017    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443028    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:15:34.443033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443034    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443048    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443055    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443071    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443077    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443085    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443091    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443099    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443105    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443119    5244 cpu_manager.go:414] "containerMap: removing container" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443124    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443142    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443148    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443160    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443166    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443185    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443191    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443283    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443293    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443301    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443310    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443318    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443345    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443356    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443365    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443375    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443383    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443391    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443401    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443408    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443416    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443424    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443436    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="50da3769-6de9-4b39-b217-3da2b24d98ff" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443443    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:15:34.443588 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.443450    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:15:34.449123 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.449102    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.553374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.549749    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d7b12e-openshift-default-dockercfg-pp8q7\""
Nov 03 22:15:34.569366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.565640    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"]
Nov 03 22:15:34.649913 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.649785    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.650186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.650168    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.650404 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.650391    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jglfk\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.650626 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.650612    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.650894 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.650881    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.651101 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.651086    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.651400 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.651380    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.651563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.651547    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.651726 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.651709    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752487 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752397    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752487 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752440    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglfk\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752487 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752482    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752554    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752593    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752675    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752710    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.752783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752757    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.753045 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752795    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.753045 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.752891    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.753158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.753053    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.753255 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.753231    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.753347 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.753293    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.753601 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.753577    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.755629 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.755552    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.755629 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.755584    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.809482 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.809440    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:34.809655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:34.809546    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jglfk\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk\") pod \"e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") " pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:35.068175 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:35.068083    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:35.501115 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:35.501085    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"]
Nov 03 22:15:35.503791 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:15:35.503752    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3be1e24_014c_4657_b8a7_2a277e135696.slice/crio-b3e7706f1b29a6b4ed389c21b165f9c65c76b3d32c9063d69c419089891db717 WatchSource:0}: Error finding container b3e7706f1b29a6b4ed389c21b165f9c65c76b3d32c9063d69c419089891db717: Status 404 returned error can't find the container with id b3e7706f1b29a6b4ed389c21b165f9c65c76b3d32c9063d69c419089891db717
Nov 03 22:15:35.507233 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:35.507190    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:15:35.507315 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:35.507287    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:15:35.507388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:35.507346    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:15:35.561835 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:35.561801    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" event={"ID":"b3be1e24-014c-4657-b8a7-2a277e135696","Type":"ContainerStarted","Data":"b3e7706f1b29a6b4ed389c21b165f9c65c76b3d32c9063d69c419089891db717"}
Nov 03 22:15:36.569165 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:36.569084    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" event={"ID":"b3be1e24-014c-4657-b8a7-2a277e135696","Type":"ContainerStarted","Data":"f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"}
Nov 03 22:15:36.738296 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:36.738245    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podStartSLOduration=2.738230916 podStartE2EDuration="2.738230916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:15:36.735296995 +0000 UTC m=+3607.719293597" watchObservedRunningTime="2026-11-03 22:15:36.738230916 +0000 UTC m=+3607.722227474"
Nov 03 22:15:37.068800 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:37.068764    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:37.074885 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:37.074859    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:37.575051 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:37.575022    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:37.576078 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:15:37.576052    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:15:52.436208 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:15:52.436166    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:16:02.963998 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:02.963958    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:16:22.437719 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:16:22.437681    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:16:30.673764 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:30.673733    5244 scope.go:122] "RemoveContainer" containerID="6ea5dd55a23b4de4401b673b28f4cb7aa8ba413bc1ae8f5ed82e92eab9db2916"
Nov 03 22:16:30.683267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:30.683244    5244 scope.go:122] "RemoveContainer" containerID="7b482bae56e093815c9075846691054031099f9bdc1897a3706df8d0436323f1"
Nov 03 22:16:46.011949 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:46.011914    5244 generic.go:356] "Generic (PLEG): container finished" podID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerID="7f5afe69b2abc43ba5a9cc426c4f6583af90dc9905895091a1d486a3d84fed72" exitCode=0
Nov 03 22:16:46.012480 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:46.011996    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" event={"ID":"21e008d8-348f-4b8a-9f7a-23066154dad2","Type":"ContainerDied","Data":"7f5afe69b2abc43ba5a9cc426c4f6583af90dc9905895091a1d486a3d84fed72"}
Nov 03 22:16:46.012571 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:46.012483    5244 scope.go:122] "RemoveContainer" containerID="7f5afe69b2abc43ba5a9cc426c4f6583af90dc9905895091a1d486a3d84fed72"
Nov 03 22:16:46.014161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:46.014133    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:16:46.014224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:46.014205    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:16:46.014261 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:46.014240    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:16:47.019651 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:47.019617    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" event={"ID":"21e008d8-348f-4b8a-9f7a-23066154dad2","Type":"ContainerStarted","Data":"4bb9fb7e68faaf0af2257e9b57fa56701b8a123d82cbaf7111f15dc3466a6d27"}
Nov 03 22:16:47.998932 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:47.998886    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:16:47.999181 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:47.999155    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 22:16:47.999273 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:47.999211    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 22:16:48.999388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:48.999312    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 22:16:48.999828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:48.999438    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 22:16:49.999142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:49.999103    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 22:16:49.999373 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:49.999169    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 22:16:50.999827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:50.999791    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 22:16:51.000370 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:50.999894    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 22:16:51.999817 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:51.999781    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 22:16:52.000083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:51.999838    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 22:16:52.436799 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:16:52.436763    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:16:53.004308 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:53.004273    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:16:53.004837 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:53.004746    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:16:53.005593 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:53.005570    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:16:57.673851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:16:57.673814    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:17:04.272469 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272432    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"]
Nov 03 22:17:04.272969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272907    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:17:04.272969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272918    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6fb05c-594a-4e9a-bf73-e241622d52e0" containerName="cleanup"
Nov 03 22:17:04.272969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272937    5244 cpu_manager.go:414] "containerMap: removing container" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:17:04.272969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272942    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:17:04.272969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272956    5244 cpu_manager.go:414] "containerMap: removing container" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:17:04.272969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272962    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272985    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272990    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.272997    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273003    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273073    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a77b83c-515e-415a-b375-0c72b0abfb85" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273083    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbe2af4-e992-497b-a95a-d672a29c842e" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273089    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09a28fe-6665-44c3-90fa-c3dc9035b301" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273095    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee72c2b-3938-457a-98a3-565450f0c7ed" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273100    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="499fbb98-6d07-4117-8dd1-99178f6fbcc3" containerName="cleanup"
Nov 03 22:17:04.273168 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.273106    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2652d3bb-1745-4a5e-8921-c84792ec9f34" containerName="cleanup"
Nov 03 22:17:04.276403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.276384    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.315952 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.315913    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnrk\" (UniqueName: \"kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.316165 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.315985    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.316165 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.316094    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.331601 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.331565    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:17:04.331782 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.331707    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:17:04.334349 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.334304    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"]
Nov 03 22:17:04.417011 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.416976    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.417011 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.417012    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.417313 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.417119    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnrk\" (UniqueName: \"kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.417750 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.417728    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.419589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.419566    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.473397 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.473366    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnrk\" (UniqueName: \"kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk\") pod \"maas-api-revoke-sub-f1da14cb-j45x4\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.589550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.589454    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:04.964539 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:04.964513    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"]
Nov 03 22:17:04.966228 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:17:04.966203    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5837fcb5_7b15_4e6a_bda0_0e423fa72963.slice/crio-4588c3cb387614783125d0890b100d2124783f665291e384dd6c8b3999b159ed WatchSource:0}: Error finding container 4588c3cb387614783125d0890b100d2124783f665291e384dd6c8b3999b159ed: Status 404 returned error can't find the container with id 4588c3cb387614783125d0890b100d2124783f665291e384dd6c8b3999b159ed
Nov 03 22:17:05.161341 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:05.161304    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4" event={"ID":"5837fcb5-7b15-4e6a-bda0-0e423fa72963","Type":"ContainerStarted","Data":"4588c3cb387614783125d0890b100d2124783f665291e384dd6c8b3999b159ed"}
Nov 03 22:17:06.169386 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:06.169254    5244 generic.go:356] "Generic (PLEG): container finished" podID="5837fcb5-7b15-4e6a-bda0-0e423fa72963" containerID="abda7eb1bf2773715e8b86bd470fcc8f958d9cc34b058b578cdc50e1efc4d650" exitCode=0
Nov 03 22:17:06.169386 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:06.169359    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4" event={"ID":"5837fcb5-7b15-4e6a-bda0-0e423fa72963","Type":"ContainerDied","Data":"abda7eb1bf2773715e8b86bd470fcc8f958d9cc34b058b578cdc50e1efc4d650"}
Nov 03 22:17:07.323594 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.323565    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:07.452609 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.452560    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk\" (UniqueName: \"kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk\") pod \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") "
Nov 03 22:17:07.452826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.452686    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca\") pod \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") "
Nov 03 22:17:07.452826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.452787    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca\") pod \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\" (UID: \"5837fcb5-7b15-4e6a-bda0-0e423fa72963\") "
Nov 03 22:17:07.453563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.453536    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca" pod "5837fcb5-7b15-4e6a-bda0-0e423fa72963" (UID: "5837fcb5-7b15-4e6a-bda0-0e423fa72963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:17:07.455277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.455248    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk" pod "5837fcb5-7b15-4e6a-bda0-0e423fa72963" (UID: "5837fcb5-7b15-4e6a-bda0-0e423fa72963"). InnerVolumeSpecName "kube-api-access-glnrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:07.455429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.455314    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca" pod "5837fcb5-7b15-4e6a-bda0-0e423fa72963" (UID: "5837fcb5-7b15-4e6a-bda0-0e423fa72963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:17:07.554500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.554460    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:07.554500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.554491    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5837fcb5-7b15-4e6a-bda0-0e423fa72963-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:07.554500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:07.554502    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glnrk\" (UniqueName: \"kubernetes.io/projected/5837fcb5-7b15-4e6a-bda0-0e423fa72963-kube-api-access-glnrk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:08.183114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:08.183079    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4" event={"ID":"5837fcb5-7b15-4e6a-bda0-0e423fa72963","Type":"ContainerDied","Data":"4588c3cb387614783125d0890b100d2124783f665291e384dd6c8b3999b159ed"}
Nov 03 22:17:08.183114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:08.183105    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"
Nov 03 22:17:08.183365 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:08.183136    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4588c3cb387614783125d0890b100d2124783f665291e384dd6c8b3999b159ed"
Nov 03 22:17:15.004837 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.004799    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"]
Nov 03 22:17:15.005309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.005295    5244 cpu_manager.go:414] "containerMap: removing container" podUID="5837fcb5-7b15-4e6a-bda0-0e423fa72963" containerName="revoke-keys"
Nov 03 22:17:15.005374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.005311    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5837fcb5-7b15-4e6a-bda0-0e423fa72963" containerName="revoke-keys"
Nov 03 22:17:15.005457 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.005445    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="5837fcb5-7b15-4e6a-bda0-0e423fa72963" containerName="revoke-keys"
Nov 03 22:17:15.008925 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.008905    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.062640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.062602    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:17:15.065714 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.065681    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"]
Nov 03 22:17:15.111902 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.111871    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:17:15.157056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.157015    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.157264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.157094    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2tw\" (UniqueName: \"kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.157264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.157163    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.258826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.258732    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.258826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.258793    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2tw\" (UniqueName: \"kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.259072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.258845    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.259701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.259674    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.261616 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.261590    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.315649 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.315619    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2tw\" (UniqueName: \"kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw\") pod \"maas-api-revoke-sub-22b4f0a4-4mq2s\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.322559 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.322537    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:15.699072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:15.699047    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"]
Nov 03 22:17:15.701583 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:17:15.701554    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1dbf58_c446_42c2_9690_c7208af622b0.slice/crio-73ad1bbbf48e0df80699478d227ac9d440ff47d411e3ee322049e68d14a3b21a WatchSource:0}: Error finding container 73ad1bbbf48e0df80699478d227ac9d440ff47d411e3ee322049e68d14a3b21a: Status 404 returned error can't find the container with id 73ad1bbbf48e0df80699478d227ac9d440ff47d411e3ee322049e68d14a3b21a
Nov 03 22:17:16.230119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:16.230017    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s" event={"ID":"2f1dbf58-c446-42c2-9690-c7208af622b0","Type":"ContainerStarted","Data":"7902fa0a9bf85033510ddf234fa8523fdfbf3419defed916957c8f52a0647bef"}
Nov 03 22:17:16.230119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:16.230082    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s" event={"ID":"2f1dbf58-c446-42c2-9690-c7208af622b0","Type":"ContainerStarted","Data":"73ad1bbbf48e0df80699478d227ac9d440ff47d411e3ee322049e68d14a3b21a"}
Nov 03 22:17:16.342230 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:16.342162    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s" podStartSLOduration=2.34214402 podStartE2EDuration="2.34214402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:16.340214053 +0000 UTC m=+3707.324210633" watchObservedRunningTime="2026-11-03 22:17:16.34214402 +0000 UTC m=+3707.326140578"
Nov 03 22:17:17.237614 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:17.237575    5244 generic.go:356] "Generic (PLEG): container finished" podID="2f1dbf58-c446-42c2-9690-c7208af622b0" containerID="7902fa0a9bf85033510ddf234fa8523fdfbf3419defed916957c8f52a0647bef" exitCode=0
Nov 03 22:17:17.238064 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:17.237667    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s" event={"ID":"2f1dbf58-c446-42c2-9690-c7208af622b0","Type":"ContainerDied","Data":"7902fa0a9bf85033510ddf234fa8523fdfbf3419defed916957c8f52a0647bef"}
Nov 03 22:17:18.392095 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.392062    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:18.497264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.497166    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca\") pod \"2f1dbf58-c446-42c2-9690-c7208af622b0\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") "
Nov 03 22:17:18.497264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.497263    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw\" (UniqueName: \"kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw\") pod \"2f1dbf58-c446-42c2-9690-c7208af622b0\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") "
Nov 03 22:17:18.497513 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.497427    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca\") pod \"2f1dbf58-c446-42c2-9690-c7208af622b0\" (UID: \"2f1dbf58-c446-42c2-9690-c7208af622b0\") "
Nov 03 22:17:18.497575 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.497551    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca" pod "2f1dbf58-c446-42c2-9690-c7208af622b0" (UID: "2f1dbf58-c446-42c2-9690-c7208af622b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:17:18.497765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.497745    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:18.499470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.499446    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw" pod "2f1dbf58-c446-42c2-9690-c7208af622b0" (UID: "2f1dbf58-c446-42c2-9690-c7208af622b0"). InnerVolumeSpecName "kube-api-access-2j2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:18.499581 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.499553    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca" pod "2f1dbf58-c446-42c2-9690-c7208af622b0" (UID: "2f1dbf58-c446-42c2-9690-c7208af622b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:17:18.599166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.599124    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2tw\" (UniqueName: \"kubernetes.io/projected/2f1dbf58-c446-42c2-9690-c7208af622b0-kube-api-access-2j2tw\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:18.599166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:18.599161    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f1dbf58-c446-42c2-9690-c7208af622b0-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:19.251076 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:19.251043    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"
Nov 03 22:17:19.251257 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:19.251086    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s" event={"ID":"2f1dbf58-c446-42c2-9690-c7208af622b0","Type":"ContainerDied","Data":"73ad1bbbf48e0df80699478d227ac9d440ff47d411e3ee322049e68d14a3b21a"}
Nov 03 22:17:19.251257 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:19.251131    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ad1bbbf48e0df80699478d227ac9d440ff47d411e3ee322049e68d14a3b21a"
Nov 03 22:17:22.437777 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:17:22.437732    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:17:28.971264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:28.971221    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"]
Nov 03 22:17:33.536384 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:33.536322    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:17:43.992696 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:43.992586    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" podUID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" containerName="main" containerID="cri-o://ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae" gracePeriod=45
Nov 03 22:17:44.162221 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.162183    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" podUID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": dial tcp 10.133.0.154:8000: connect: connection refused"
Nov 03 22:17:44.251781 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.251756    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:17:44.330612 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330574    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache\") pod \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") "
Nov 03 22:17:44.330829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330640    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw\" (UniqueName: \"kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw\") pod \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") "
Nov 03 22:17:44.330829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330678    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir\") pod \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") "
Nov 03 22:17:44.330829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330708    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs\") pod \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") "
Nov 03 22:17:44.330829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330784    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm\") pod \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") "
Nov 03 22:17:44.330829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330804    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home\") pod \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\" (UID: \"4c781c17-58fd-41a3-9cc5-c779c23c4f65\") "
Nov 03 22:17:44.331124 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330867    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache" pod "4c781c17-58fd-41a3-9cc5-c779c23c4f65" (UID: "4c781c17-58fd-41a3-9cc5-c779c23c4f65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:44.331124 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.330992    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir" pod "4c781c17-58fd-41a3-9cc5-c779c23c4f65" (UID: "4c781c17-58fd-41a3-9cc5-c779c23c4f65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:44.331249 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.331195    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home" pod "4c781c17-58fd-41a3-9cc5-c779c23c4f65" (UID: "4c781c17-58fd-41a3-9cc5-c779c23c4f65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:44.331249 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.331215    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:44.331249 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.331233    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:44.333231 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.333204    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm" pod "4c781c17-58fd-41a3-9cc5-c779c23c4f65" (UID: "4c781c17-58fd-41a3-9cc5-c779c23c4f65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:17:44.333363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.333230    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs" pod "4c781c17-58fd-41a3-9cc5-c779c23c4f65" (UID: "4c781c17-58fd-41a3-9cc5-c779c23c4f65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:17:44.333363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.333307    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw" pod "4c781c17-58fd-41a3-9cc5-c779c23c4f65" (UID: "4c781c17-58fd-41a3-9cc5-c779c23c4f65"). InnerVolumeSpecName "kube-api-access-2ssbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:44.424484 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.424389    5244 generic.go:356] "Generic (PLEG): container finished" podID="b3be1e24-014c-4657-b8a7-2a277e135696" containerID="f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51" exitCode=0
Nov 03 22:17:44.424484 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.424445    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" event={"ID":"b3be1e24-014c-4657-b8a7-2a277e135696","Type":"ContainerDied","Data":"f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"}
Nov 03 22:17:44.424892 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.424871    5244 scope.go:122] "RemoveContainer" containerID="f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"
Nov 03 22:17:44.426034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426007    5244 generic.go:356] "Generic (PLEG): container finished" podID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" containerID="ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae" exitCode=0
Nov 03 22:17:44.426162 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426149    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" event={"ID":"4c781c17-58fd-41a3-9cc5-c779c23c4f65","Type":"ContainerDied","Data":"ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae"}
Nov 03 22:17:44.426216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426174    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4" event={"ID":"4c781c17-58fd-41a3-9cc5-c779c23c4f65","Type":"ContainerDied","Data":"b4842da76b8e721a74e8af506878fe64f83a39bc58a4a4d879c8b41dbebd63ec"}
Nov 03 22:17:44.426216 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426190    5244 scope.go:122] "RemoveContainer" containerID="ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae"
Nov 03 22:17:44.426339 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426312    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"
Nov 03 22:17:44.426627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426593    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:17:44.426702 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426676    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:17:44.426763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.426730    5244 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Nov 03 22:17:44.432293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.432070    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ssbw\" (UniqueName: \"kubernetes.io/projected/4c781c17-58fd-41a3-9cc5-c779c23c4f65-kube-api-access-2ssbw\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:44.432293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.432101    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c781c17-58fd-41a3-9cc5-c779c23c4f65-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:44.432293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.432117    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:44.432293 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.432130    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c781c17-58fd-41a3-9cc5-c779c23c4f65-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:44.441075 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.441046    5244 scope.go:122] "RemoveContainer" containerID="ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae"
Nov 03 22:17:44.442077 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:17:44.442049    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae\": container with ID starting with ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae not found: ID does not exist" containerID="ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae"
Nov 03 22:17:44.442176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.442087    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae"} err="failed to get container status \"ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae\": rpc error: code = NotFound desc = could not find container \"ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae\": container with ID starting with ff741dfb01f49fc775c10a316f54d3173d0a99aebe69aef6334520e2339e6fae not found: ID does not exist"
Nov 03 22:17:44.658618 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.658563    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"]
Nov 03 22:17:44.661041 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:44.661013    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e15bbc/auth-test-model-e15bbc-kserve-798b45bb5c-dgfs4"]
Nov 03 22:17:45.144407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:45.139745    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" path="/var/lib/kubelet/pods/4c781c17-58fd-41a3-9cc5-c779c23c4f65/volumes"
Nov 03 22:17:45.446353 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:45.446089    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" event={"ID":"b3be1e24-014c-4657-b8a7-2a277e135696","Type":"ContainerStarted","Data":"9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61"}
Nov 03 22:17:46.069251 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:46.069209    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:17:46.069490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:46.069465    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 22:17:46.069555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:46.069511    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 22:17:47.069145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:47.069111    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 22:17:47.069596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:47.069165    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 22:17:48.068545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:48.068501    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 22:17:48.068778 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:48.068564    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 22:17:49.069310 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:49.069279    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 22:17:49.069730 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:49.069372    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 22:17:50.069109 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:50.069065    5244 patch_prober.go:29] interesting pod/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 22:17:50.069384 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:50.069135    5244 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 22:17:51.073926 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:51.073897    5244 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:17:51.074434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:51.074188    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:17:51.074929 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:51.074913    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:17:52.437672 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:17:52.437630    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:17:55.143733 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.143696    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"]
Nov 03 22:17:55.144392 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.144372    5244 cpu_manager.go:414] "containerMap: removing container" podUID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" containerName="main"
Nov 03 22:17:55.144490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.144394    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" containerName="main"
Nov 03 22:17:55.144490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.144414    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2f1dbf58-c446-42c2-9690-c7208af622b0" containerName="revoke-keys"
Nov 03 22:17:55.144490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.144421    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1dbf58-c446-42c2-9690-c7208af622b0" containerName="revoke-keys"
Nov 03 22:17:55.144657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.144532    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c781c17-58fd-41a3-9cc5-c779c23c4f65" containerName="main"
Nov 03 22:17:55.144657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.144549    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1dbf58-c446-42c2-9690-c7208af622b0" containerName="revoke-keys"
Nov 03 22:17:55.148978 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.148957    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.251053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.251014    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:17:55.251213 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.251115    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:17:55.253607 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.253587    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"]
Nov 03 22:17:55.298463 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.298430    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hrr\" (UniqueName: \"kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.298640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.298542    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.298685 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.298668    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.399569 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.399471    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.399569 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.399561    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.399824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.399619    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hrr\" (UniqueName: \"kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.400281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.400253    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.402188 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.402160    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.456870 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.456839    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hrr\" (UniqueName: \"kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr\") pod \"maas-api-revoke-keys-e2e-extmod-a-288188-cn76p\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.462718 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.462687    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:55.838244 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:55.838218    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"]
Nov 03 22:17:55.840187 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:17:55.840155    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047b9a3c_940e_4e74_835b_a23909641849.slice/crio-a5d865a3a871ef193e797a4fb8a893f0829cd6e0497b8f149aaaa1318d25807e WatchSource:0}: Error finding container a5d865a3a871ef193e797a4fb8a893f0829cd6e0497b8f149aaaa1318d25807e: Status 404 returned error can't find the container with id a5d865a3a871ef193e797a4fb8a893f0829cd6e0497b8f149aaaa1318d25807e
Nov 03 22:17:56.538315 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:56.538280    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p" event={"ID":"047b9a3c-940e-4e74-835b-a23909641849","Type":"ContainerStarted","Data":"46e5e5e17fd5a38afb6aa2141cbfafb03bd81c58a2a2dff18bed817ef6e208c3"}
Nov 03 22:17:56.538864 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:56.538350    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p" event={"ID":"047b9a3c-940e-4e74-835b-a23909641849","Type":"ContainerStarted","Data":"a5d865a3a871ef193e797a4fb8a893f0829cd6e0497b8f149aaaa1318d25807e"}
Nov 03 22:17:56.652129 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:56.652074    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p" podStartSLOduration=1.652058378 podStartE2EDuration="1.652058378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:56.650253249 +0000 UTC m=+3747.634249829" watchObservedRunningTime="2026-11-03 22:17:56.652058378 +0000 UTC m=+3747.636054959"
Nov 03 22:17:57.545558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:57.545520    5244 generic.go:356] "Generic (PLEG): container finished" podID="047b9a3c-940e-4e74-835b-a23909641849" containerID="46e5e5e17fd5a38afb6aa2141cbfafb03bd81c58a2a2dff18bed817ef6e208c3" exitCode=0
Nov 03 22:17:57.545973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:57.545607    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p" event={"ID":"047b9a3c-940e-4e74-835b-a23909641849","Type":"ContainerDied","Data":"46e5e5e17fd5a38afb6aa2141cbfafb03bd81c58a2a2dff18bed817ef6e208c3"}
Nov 03 22:17:58.683609 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.683581    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:58.841734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.841638    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca\") pod \"047b9a3c-940e-4e74-835b-a23909641849\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") "
Nov 03 22:17:58.841734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.841706    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr\" (UniqueName: \"kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr\") pod \"047b9a3c-940e-4e74-835b-a23909641849\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") "
Nov 03 22:17:58.841968 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.841760    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca\") pod \"047b9a3c-940e-4e74-835b-a23909641849\" (UID: \"047b9a3c-940e-4e74-835b-a23909641849\") "
Nov 03 22:17:58.842208 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.842169    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca" pod "047b9a3c-940e-4e74-835b-a23909641849" (UID: "047b9a3c-940e-4e74-835b-a23909641849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:17:58.844224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.844192    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca" pod "047b9a3c-940e-4e74-835b-a23909641849" (UID: "047b9a3c-940e-4e74-835b-a23909641849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:17:58.844555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.844530    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr" pod "047b9a3c-940e-4e74-835b-a23909641849" (UID: "047b9a3c-940e-4e74-835b-a23909641849"). InnerVolumeSpecName "kube-api-access-g6hrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:17:58.943083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.943050    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047b9a3c-940e-4e74-835b-a23909641849-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:58.943083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.943081    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hrr\" (UniqueName: \"kubernetes.io/projected/047b9a3c-940e-4e74-835b-a23909641849-kube-api-access-g6hrr\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:58.943083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:58.943093    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047b9a3c-940e-4e74-835b-a23909641849-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:17:59.558736 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:59.558704    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"
Nov 03 22:17:59.558736 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:59.558720    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p" event={"ID":"047b9a3c-940e-4e74-835b-a23909641849","Type":"ContainerDied","Data":"a5d865a3a871ef193e797a4fb8a893f0829cd6e0497b8f149aaaa1318d25807e"}
Nov 03 22:17:59.558958 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:17:59.558764    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d865a3a871ef193e797a4fb8a893f0829cd6e0497b8f149aaaa1318d25807e"
Nov 03 22:18:06.658281 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:06.658249    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"]
Nov 03 22:18:06.766156 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:06.766113    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"]
Nov 03 22:18:11.008343 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.008301    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"]
Nov 03 22:18:11.008860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.008825    5244 cpu_manager.go:414] "containerMap: removing container" podUID="047b9a3c-940e-4e74-835b-a23909641849" containerName="revoke-keys"
Nov 03 22:18:11.008860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.008840    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047b9a3c-940e-4e74-835b-a23909641849" containerName="revoke-keys"
Nov 03 22:18:11.008965 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.008952    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="047b9a3c-940e-4e74-835b-a23909641849" containerName="revoke-keys"
Nov 03 22:18:11.012289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.012271    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.069648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.069614    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"]
Nov 03 22:18:11.117354 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.103925    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.141355 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.117748    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nblj\" (UniqueName: \"kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj\") pod \"payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.175871 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.142743    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-48ae0a\""
Nov 03 22:18:11.177114 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.142838    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-48ae0a-dockercfg-rf4l9\""
Nov 03 22:18:11.180498 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.180474    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"]
Nov 03 22:18:11.184819 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.184796    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.221998 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.221959    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"]
Nov 03 22:18:11.276688 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.276579    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nblj\" (UniqueName: \"kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj\") pod \"payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.276939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.276717    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.276939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.276753    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.276939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.276779    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnl8\" (UniqueName: \"kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8\") pod \"payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.277522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.277501    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.334017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.333983    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nblj\" (UniqueName: \"kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj\") pod \"payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.377734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.377696    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnl8\" (UniqueName: \"kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8\") pod \"payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.377905 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.377860    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.378600 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.378578    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.433948 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.433902    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnl8\" (UniqueName: \"kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8\") pod \"payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.498572 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.498530    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:11.626096 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.626066    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:11.658550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.658498    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt" podUID="76ed667a-8f47-4f86-9a40-5c2b1db7600d" containerName="payload-processing" containerID="cri-o://1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0" gracePeriod=25
Nov 03 22:18:11.767236 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.767189    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk" podUID="6d7ace1d-a569-46e7-bc8a-36428bc956f6" containerName="payload-pre-processing" containerID="cri-o://703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660" gracePeriod=25
Nov 03 22:18:11.877587 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.875983    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"]
Nov 03 22:18:11.877587 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:18:11.877054    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414b4812_e494_43ed_aecf_0dccdc023f0f.slice/crio-a01fb5c2eefb6c85f27278d3a2a81e173309b814f2785cf94de9ff2b30fa83a7 WatchSource:0}: Error finding container a01fb5c2eefb6c85f27278d3a2a81e173309b814f2785cf94de9ff2b30fa83a7: Status 404 returned error can't find the container with id a01fb5c2eefb6c85f27278d3a2a81e173309b814f2785cf94de9ff2b30fa83a7
Nov 03 22:18:11.928867 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:11.928831    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:18:12.003163 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.002954    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"]
Nov 03 22:18:12.009410 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:18:12.009376    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515dcc36_e0a4_4e79_9e1c_3d5544003ab7.slice/crio-dc499d0eb7ae8f30550455955943ba45a984fc40de20949e9b5e05f57721447d WatchSource:0}: Error finding container dc499d0eb7ae8f30550455955943ba45a984fc40de20949e9b5e05f57721447d: Status 404 returned error can't find the container with id dc499d0eb7ae8f30550455955943ba45a984fc40de20949e9b5e05f57721447d
Nov 03 22:18:12.032496 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.032475    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:18:12.086759 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.086683    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr\" (UniqueName: \"kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr\") pod \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") "
Nov 03 22:18:12.086881 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.086762    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume\") pod \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\" (UID: \"76ed667a-8f47-4f86-9a40-5c2b1db7600d\") "
Nov 03 22:18:12.087316 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.087288    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume" pod "76ed667a-8f47-4f86-9a40-5c2b1db7600d" (UID: "76ed667a-8f47-4f86-9a40-5c2b1db7600d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:18:12.088961 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.088936    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr" pod "76ed667a-8f47-4f86-9a40-5c2b1db7600d" (UID: "76ed667a-8f47-4f86-9a40-5c2b1db7600d"). InnerVolumeSpecName "kube-api-access-xdfmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:18:12.188727 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.188677    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume\") pod \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") "
Nov 03 22:18:12.188898 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.188808    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q\" (UniqueName: \"kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q\") pod \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\" (UID: \"6d7ace1d-a569-46e7-bc8a-36428bc956f6\") "
Nov 03 22:18:12.189072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.189048    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume" pod "6d7ace1d-a569-46e7-bc8a-36428bc956f6" (UID: "6d7ace1d-a569-46e7-bc8a-36428bc956f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:18:12.189267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.189248    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d7ace1d-a569-46e7-bc8a-36428bc956f6-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:12.189267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.189266    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfmr\" (UniqueName: \"kubernetes.io/projected/76ed667a-8f47-4f86-9a40-5c2b1db7600d-kube-api-access-xdfmr\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:12.189484 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.189276    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ed667a-8f47-4f86-9a40-5c2b1db7600d-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:12.191033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.191005    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q" pod "6d7ace1d-a569-46e7-bc8a-36428bc956f6" (UID: "6d7ace1d-a569-46e7-bc8a-36428bc956f6"). InnerVolumeSpecName "kube-api-access-mp95q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:18:12.290427 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.290384    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp95q\" (UniqueName: \"kubernetes.io/projected/6d7ace1d-a569-46e7-bc8a-36428bc956f6-kube-api-access-mp95q\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:12.638595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.638465    5244 generic.go:356] "Generic (PLEG): container finished" podID="6d7ace1d-a569-46e7-bc8a-36428bc956f6" containerID="703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660" exitCode=0
Nov 03 22:18:12.638595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.638515    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"
Nov 03 22:18:12.638595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.638529    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk" event={"ID":"6d7ace1d-a569-46e7-bc8a-36428bc956f6","Type":"ContainerDied","Data":"703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660"}
Nov 03 22:18:12.638595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.638594    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk" event={"ID":"6d7ace1d-a569-46e7-bc8a-36428bc956f6","Type":"ContainerDied","Data":"efd91745b4a062dff23c02105dfe5eab78cec52bdbf5386cfd7c6d6f9f9069b0"}
Nov 03 22:18:12.638994 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.638627    5244 scope.go:122] "RemoveContainer" containerID="703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660"
Nov 03 22:18:12.645632 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.645598    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr" event={"ID":"414b4812-e494-43ed-aecf-0dccdc023f0f","Type":"ContainerStarted","Data":"18292c7297c59e72df15d1cfb4c2de50920f141339b2def439ad7ae667cb4f39"}
Nov 03 22:18:12.645787 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.645669    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:12.645787 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.645691    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr" event={"ID":"414b4812-e494-43ed-aecf-0dccdc023f0f","Type":"ContainerStarted","Data":"a01fb5c2eefb6c85f27278d3a2a81e173309b814f2785cf94de9ff2b30fa83a7"}
Nov 03 22:18:12.647692 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.647667    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79" event={"ID":"515dcc36-e0a4-4e79-9e1c-3d5544003ab7","Type":"ContainerStarted","Data":"e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c"}
Nov 03 22:18:12.647829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.647716    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79" event={"ID":"515dcc36-e0a4-4e79-9e1c-3d5544003ab7","Type":"ContainerStarted","Data":"dc499d0eb7ae8f30550455955943ba45a984fc40de20949e9b5e05f57721447d"}
Nov 03 22:18:12.647829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.647745    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:12.649686 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.649660    5244 generic.go:356] "Generic (PLEG): container finished" podID="76ed667a-8f47-4f86-9a40-5c2b1db7600d" containerID="1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0" exitCode=0
Nov 03 22:18:12.649794 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.649714    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"
Nov 03 22:18:12.649794 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.649770    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt" event={"ID":"76ed667a-8f47-4f86-9a40-5c2b1db7600d","Type":"ContainerDied","Data":"1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0"}
Nov 03 22:18:12.649906 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.649816    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt" event={"ID":"76ed667a-8f47-4f86-9a40-5c2b1db7600d","Type":"ContainerDied","Data":"88d3c9fca5193681319c5054d0a0b7f345c05962bbf1fcd8f66070c6f11c19f1"}
Nov 03 22:18:12.650809 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.650790    5244 scope.go:122] "RemoveContainer" containerID="703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660"
Nov 03 22:18:12.651084 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:18:12.651064    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660\": container with ID starting with 703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660 not found: ID does not exist" containerID="703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660"
Nov 03 22:18:12.651142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.651093    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660"} err="failed to get container status \"703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660\": rpc error: code = NotFound desc = could not find container \"703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660\": container with ID starting with 703c14f012243773be7e3525ea19168f1a9be4ab33467bfc46b94758a52bf660 not found: ID does not exist"
Nov 03 22:18:12.651142 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.651110    5244 scope.go:122] "RemoveContainer" containerID="1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0"
Nov 03 22:18:12.660125 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.660107    5244 scope.go:122] "RemoveContainer" containerID="1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0"
Nov 03 22:18:12.660457 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:18:12.660437    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0\": container with ID starting with 1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0 not found: ID does not exist" containerID="1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0"
Nov 03 22:18:12.660539 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.660464    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0"} err="failed to get container status \"1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0\": rpc error: code = NotFound desc = could not find container \"1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0\": container with ID starting with 1acf7b65b01d69a3d0f830f9950d8f7324dd7b62d2d5fe58bc00b78d432e1ce0 not found: ID does not exist"
Nov 03 22:18:12.804448 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.804393    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr" podStartSLOduration=1.804377833 podStartE2EDuration="1.804377833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:12.802481856 +0000 UTC m=+3763.786478416" watchObservedRunningTime="2026-11-03 22:18:12.804377833 +0000 UTC m=+3763.788374389"
Nov 03 22:18:12.935691 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.935658    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"]
Nov 03 22:18:12.939268 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:12.939235    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-288188-849858c758-9nctt"]
Nov 03 22:18:13.044287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:13.044218    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79" podStartSLOduration=3.044200618 podStartE2EDuration="3.044200618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:13.043574587 +0000 UTC m=+3764.027571139" watchObservedRunningTime="2026-11-03 22:18:13.044200618 +0000 UTC m=+3764.028197177"
Nov 03 22:18:13.135985 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:13.135948    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ed667a-8f47-4f86-9a40-5c2b1db7600d" path="/var/lib/kubelet/pods/76ed667a-8f47-4f86-9a40-5c2b1db7600d/volumes"
Nov 03 22:18:13.156440 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:13.156374    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"]
Nov 03 22:18:13.165374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:13.158540    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-288188-5f6bff9cff-l8nhk"]
Nov 03 22:18:15.136189 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:15.136158    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7ace1d-a569-46e7-bc8a-36428bc956f6" path="/var/lib/kubelet/pods/6d7ace1d-a569-46e7-bc8a-36428bc956f6/volumes"
Nov 03 22:18:17.465559 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.465522    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"]
Nov 03 22:18:17.466003 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.465989    5244 cpu_manager.go:414] "containerMap: removing container" podUID="76ed667a-8f47-4f86-9a40-5c2b1db7600d" containerName="payload-processing"
Nov 03 22:18:17.466062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.466004    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ed667a-8f47-4f86-9a40-5c2b1db7600d" containerName="payload-processing"
Nov 03 22:18:17.466062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.466020    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6d7ace1d-a569-46e7-bc8a-36428bc956f6" containerName="payload-pre-processing"
Nov 03 22:18:17.466062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.466025    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7ace1d-a569-46e7-bc8a-36428bc956f6" containerName="payload-pre-processing"
Nov 03 22:18:17.466163 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.466123    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ed667a-8f47-4f86-9a40-5c2b1db7600d" containerName="payload-processing"
Nov 03 22:18:17.466163 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.466133    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7ace1d-a569-46e7-bc8a-36428bc956f6" containerName="payload-pre-processing"
Nov 03 22:18:17.471021 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.471004    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.526522 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.526476    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"]
Nov 03 22:18:17.572389 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.572359    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:18:17.572661 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.572647    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:18:17.651249 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.651218    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjztq\" (UniqueName: \"kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.651456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.651254    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.651456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.651418    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.752779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.752694    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjztq\" (UniqueName: \"kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.752779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.752733    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.753035 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.752819    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.753472 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.753451    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.755397 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.755377    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:17.809536 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:17.809506    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjztq\" (UniqueName: \"kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:18.084157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:18.084064    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:18.456245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:18.456215    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"]
Nov 03 22:18:18.458550 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:18:18.458524    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea80619_a418_4586_980e_a18da13c2b35.slice/crio-dbbf72807c6ef1c1049f74925080162fd48d14f14d69bff598009c4475f5a304 WatchSource:0}: Error finding container dbbf72807c6ef1c1049f74925080162fd48d14f14d69bff598009c4475f5a304: Status 404 returned error can't find the container with id dbbf72807c6ef1c1049f74925080162fd48d14f14d69bff598009c4475f5a304
Nov 03 22:18:18.689313 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:18.689281    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n" event={"ID":"eea80619-a418-4586-980e-a18da13c2b35","Type":"ContainerStarted","Data":"dbbf72807c6ef1c1049f74925080162fd48d14f14d69bff598009c4475f5a304"}
Nov 03 22:18:19.696463 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:19.696421    5244 generic.go:356] "Generic (PLEG): container finished" podID="eea80619-a418-4586-980e-a18da13c2b35" containerID="469a7eea0080bdc3fca8b532cfe815a30de53b845e45fe27a13ad4cc894c9253" exitCode=0
Nov 03 22:18:19.696898 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:19.696513    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n" event={"ID":"eea80619-a418-4586-980e-a18da13c2b35","Type":"ContainerDied","Data":"469a7eea0080bdc3fca8b532cfe815a30de53b845e45fe27a13ad4cc894c9253"}
Nov 03 22:18:20.838460 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.838430    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:20.986383 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.986280    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca\") pod \"eea80619-a418-4586-980e-a18da13c2b35\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") "
Nov 03 22:18:20.986548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.986453    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca\") pod \"eea80619-a418-4586-980e-a18da13c2b35\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") "
Nov 03 22:18:20.986548 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.986490    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq\" (UniqueName: \"kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq\") pod \"eea80619-a418-4586-980e-a18da13c2b35\" (UID: \"eea80619-a418-4586-980e-a18da13c2b35\") "
Nov 03 22:18:20.986893 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.986865    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca" pod "eea80619-a418-4586-980e-a18da13c2b35" (UID: "eea80619-a418-4586-980e-a18da13c2b35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:18:20.988632 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.988595    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca" pod "eea80619-a418-4586-980e-a18da13c2b35" (UID: "eea80619-a418-4586-980e-a18da13c2b35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:18:20.988847 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:20.988826    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq" pod "eea80619-a418-4586-980e-a18da13c2b35" (UID: "eea80619-a418-4586-980e-a18da13c2b35"). InnerVolumeSpecName "kube-api-access-gjztq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:18:21.087763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:21.087654    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea80619-a418-4586-980e-a18da13c2b35-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:21.087763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:21.087720    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea80619-a418-4586-980e-a18da13c2b35-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:21.087763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:21.087738    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjztq\" (UniqueName: \"kubernetes.io/projected/eea80619-a418-4586-980e-a18da13c2b35-kube-api-access-gjztq\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:21.709510 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:21.709481    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"
Nov 03 22:18:21.709720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:21.709513    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n" event={"ID":"eea80619-a418-4586-980e-a18da13c2b35","Type":"ContainerDied","Data":"dbbf72807c6ef1c1049f74925080162fd48d14f14d69bff598009c4475f5a304"}
Nov 03 22:18:21.709720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:21.709563    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbf72807c6ef1c1049f74925080162fd48d14f14d69bff598009c4475f5a304"
Nov 03 22:18:22.436002 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:18:22.435966    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:18:23.660742 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:23.660709    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:18:23.661274 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:23.660899    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:18:30.047244 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:30.047210    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"]
Nov 03 22:18:30.096485 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:30.096456    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f7d4b-l6pvf"]
Nov 03 22:18:31.135904 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:31.135811    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d65b028-da2f-4335-a853-ce90ae9b93a1" path="/var/lib/kubelet/pods/1d65b028-da2f-4335-a853-ce90ae9b93a1/volumes"
Nov 03 22:18:34.200914 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:34.200877    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:18:38.616439 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.616360    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"]
Nov 03 22:18:38.616847 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.616834    5244 cpu_manager.go:414] "containerMap: removing container" podUID="eea80619-a418-4586-980e-a18da13c2b35" containerName="revoke-keys"
Nov 03 22:18:38.616892 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.616849    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea80619-a418-4586-980e-a18da13c2b35" containerName="revoke-keys"
Nov 03 22:18:38.616945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.616935    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea80619-a418-4586-980e-a18da13c2b35" containerName="revoke-keys"
Nov 03 22:18:38.620084 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.620065    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.688232 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.688191    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.688428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.688308    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89zx\" (UniqueName: \"kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.688545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.688521    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.723021 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.722975    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:18:38.723214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.723028    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:18:38.725823 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.725803    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"]
Nov 03 22:18:38.789921 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.789890    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.790176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.789937    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89zx\" (UniqueName: \"kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.790176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.790032    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.790591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.790568    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.792550 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.792524    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.846872 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.846841    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89zx\" (UniqueName: \"kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx\") pod \"maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:38.933696 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:38.933671    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:39.307792 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:39.307725    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"]
Nov 03 22:18:39.310419 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:18:39.310390    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06552027_4c4b_4a63_8f07_f2f72c986463.slice/crio-45be0d7d8ff93b2c295f4e99f526a2d2107be690914a8fe4c83e376294110c5f WatchSource:0}: Error finding container 45be0d7d8ff93b2c295f4e99f526a2d2107be690914a8fe4c83e376294110c5f: Status 404 returned error can't find the container with id 45be0d7d8ff93b2c295f4e99f526a2d2107be690914a8fe4c83e376294110c5f
Nov 03 22:18:39.830052 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:39.830004    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl" event={"ID":"06552027-4c4b-4a63-8f07-f2f72c986463","Type":"ContainerStarted","Data":"449cfa24f1fa03b6cdc988bb983055c7fa8738dba3b6cec588bb6577641f769a"}
Nov 03 22:18:39.830517 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:39.830075    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl" event={"ID":"06552027-4c4b-4a63-8f07-f2f72c986463","Type":"ContainerStarted","Data":"45be0d7d8ff93b2c295f4e99f526a2d2107be690914a8fe4c83e376294110c5f"}
Nov 03 22:18:39.944296 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:39.944239    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl" podStartSLOduration=1.944222917 podStartE2EDuration="1.944222917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:39.942102132 +0000 UTC m=+3790.926098708" watchObservedRunningTime="2026-11-03 22:18:39.944222917 +0000 UTC m=+3790.928219521"
Nov 03 22:18:40.846300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:40.846261    5244 generic.go:356] "Generic (PLEG): container finished" podID="06552027-4c4b-4a63-8f07-f2f72c986463" containerID="449cfa24f1fa03b6cdc988bb983055c7fa8738dba3b6cec588bb6577641f769a" exitCode=0
Nov 03 22:18:40.846743 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:40.846366    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl" event={"ID":"06552027-4c4b-4a63-8f07-f2f72c986463","Type":"ContainerDied","Data":"449cfa24f1fa03b6cdc988bb983055c7fa8738dba3b6cec588bb6577641f769a"}
Nov 03 22:18:41.044206 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:41.044174    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"]
Nov 03 22:18:41.046491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:41.046462    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3620f1b-vmkkz"]
Nov 03 22:18:41.136857 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:41.136775    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c" path="/var/lib/kubelet/pods/4c05c0ff-6fd4-4f15-bb9a-2f31f59f4c5c/volumes"
Nov 03 22:18:41.995934 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:41.995910    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:42.127131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.127049    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca\") pod \"06552027-4c4b-4a63-8f07-f2f72c986463\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") "
Nov 03 22:18:42.127131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.127119    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca\") pod \"06552027-4c4b-4a63-8f07-f2f72c986463\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") "
Nov 03 22:18:42.127398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.127171    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx\" (UniqueName: \"kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx\") pod \"06552027-4c4b-4a63-8f07-f2f72c986463\" (UID: \"06552027-4c4b-4a63-8f07-f2f72c986463\") "
Nov 03 22:18:42.127503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.127481    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca" pod "06552027-4c4b-4a63-8f07-f2f72c986463" (UID: "06552027-4c4b-4a63-8f07-f2f72c986463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:18:42.129415 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.129384    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca" pod "06552027-4c4b-4a63-8f07-f2f72c986463" (UID: "06552027-4c4b-4a63-8f07-f2f72c986463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:18:42.129574 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.129514    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx" pod "06552027-4c4b-4a63-8f07-f2f72c986463" (UID: "06552027-4c4b-4a63-8f07-f2f72c986463"). InnerVolumeSpecName "kube-api-access-m89zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:18:42.228950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.228913    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:42.228950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.228948    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89zx\" (UniqueName: \"kubernetes.io/projected/06552027-4c4b-4a63-8f07-f2f72c986463-kube-api-access-m89zx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:42.229223 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.228965    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06552027-4c4b-4a63-8f07-f2f72c986463-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:18:42.858972 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.858937    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"
Nov 03 22:18:42.859159 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.858979    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl" event={"ID":"06552027-4c4b-4a63-8f07-f2f72c986463","Type":"ContainerDied","Data":"45be0d7d8ff93b2c295f4e99f526a2d2107be690914a8fe4c83e376294110c5f"}
Nov 03 22:18:42.859159 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:18:42.859026    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45be0d7d8ff93b2c295f4e99f526a2d2107be690914a8fe4c83e376294110c5f"
Nov 03 22:18:52.436552 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:18:52.436516    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:19:12.390361 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.390310    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"]
Nov 03 22:19:12.390868 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.390770    5244 cpu_manager.go:414] "containerMap: removing container" podUID="06552027-4c4b-4a63-8f07-f2f72c986463" containerName="revoke-keys"
Nov 03 22:19:12.390868 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.390780    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06552027-4c4b-4a63-8f07-f2f72c986463" containerName="revoke-keys"
Nov 03 22:19:12.390951 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.390875    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="06552027-4c4b-4a63-8f07-f2f72c986463" containerName="revoke-keys"
Nov 03 22:19:12.394289 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.394270    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.479772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.477063    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.479772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.477147    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkc52\" (UniqueName: \"kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.479772 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.477285    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.497482 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.497448    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:19:12.497686 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.497574    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:19:12.500061 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.500039    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"]
Nov 03 22:19:12.578619 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.578576    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.578822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.578701    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.578822 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.578750    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkc52\" (UniqueName: \"kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.579381 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.579320    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.581233 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.581209    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.635725 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.635690    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkc52\" (UniqueName: \"kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52\") pod \"maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:12.708410 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:12.708374    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:13.085500 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:13.085470    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"]
Nov 03 22:19:13.087291 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:19:13.087258    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b0b9b2_531c_4021_b8cc_bd2cf1866a51.slice/crio-ce63f05df0f26e9586af5f4382ea8367275c288a0fe45f29d20344fe012fafa1 WatchSource:0}: Error finding container ce63f05df0f26e9586af5f4382ea8367275c288a0fe45f29d20344fe012fafa1: Status 404 returned error can't find the container with id ce63f05df0f26e9586af5f4382ea8367275c288a0fe45f29d20344fe012fafa1
Nov 03 22:19:14.069100 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:14.069066    5244 generic.go:356] "Generic (PLEG): container finished" podID="f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" containerID="f6af507f3411561a2ddb71a7222eb64a12386ae47ba793c05e1aa2b6317862aa" exitCode=0
Nov 03 22:19:14.069537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:14.069154    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4" event={"ID":"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51","Type":"ContainerDied","Data":"f6af507f3411561a2ddb71a7222eb64a12386ae47ba793c05e1aa2b6317862aa"}
Nov 03 22:19:14.069537 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:14.069219    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4" event={"ID":"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51","Type":"ContainerStarted","Data":"ce63f05df0f26e9586af5f4382ea8367275c288a0fe45f29d20344fe012fafa1"}
Nov 03 22:19:15.217728 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.217701    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:15.308667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.308635    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca\") pod \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") "
Nov 03 22:19:15.308868 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.308707    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52\" (UniqueName: \"kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52\") pod \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") "
Nov 03 22:19:15.308868 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.308793    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca\") pod \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\" (UID: \"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51\") "
Nov 03 22:19:15.309042 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.309012    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca" pod "f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" (UID: "f8b0b9b2-531c-4021-b8cc-bd2cf1866a51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:15.311008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.310986    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca" pod "f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" (UID: "f8b0b9b2-531c-4021-b8cc-bd2cf1866a51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:15.311090 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.311069    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52" pod "f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" (UID: "f8b0b9b2-531c-4021-b8cc-bd2cf1866a51"). InnerVolumeSpecName "kube-api-access-lkc52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:15.409737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.409648    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkc52\" (UniqueName: \"kubernetes.io/projected/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-kube-api-access-lkc52\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:15.409737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.409682    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:15.409737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:15.409692    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:16.082314 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:16.082271    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"
Nov 03 22:19:16.082314 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:16.082301    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4" event={"ID":"f8b0b9b2-531c-4021-b8cc-bd2cf1866a51","Type":"ContainerDied","Data":"ce63f05df0f26e9586af5f4382ea8367275c288a0fe45f29d20344fe012fafa1"}
Nov 03 22:19:16.082570 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:16.082370    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce63f05df0f26e9586af5f4382ea8367275c288a0fe45f29d20344fe012fafa1"
Nov 03 22:19:22.437937 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:19:22.437894    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:19:23.413638 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:23.413598    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"]
Nov 03 22:19:23.459396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:23.459357    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-288188-cn76p"]
Nov 03 22:19:25.136260 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:25.136224    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047b9a3c-940e-4e74-835b-a23909641849" path="/var/lib/kubelet/pods/047b9a3c-940e-4e74-835b-a23909641849/volumes"
Nov 03 22:19:29.796520 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:29.796481    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"]
Nov 03 22:19:29.845659 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:29.845624    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5b1489-dmq4n"]
Nov 03 22:19:30.004702 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:30.004663    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"]
Nov 03 22:19:30.004966 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:30.004931    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy" containerID="cri-o://4bb9fb7e68faaf0af2257e9b57fa56701b8a123d82cbaf7111f15dc3466a6d27" gracePeriod=30
Nov 03 22:19:30.044606 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:30.044564    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"]
Nov 03 22:19:30.047798 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:30.047722    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfd609ae-gmfzd"]
Nov 03 22:19:30.768166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:30.768126    5244 scope.go:122] "RemoveContainer" containerID="341ac53dd3316f904af603f1295313596176440c14d8b0df5eb42a4c8c116488"
Nov 03 22:19:31.136524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:31.136441    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a512b18-492c-4a74-9dd0-6102427a57d3" path="/var/lib/kubelet/pods/0a512b18-492c-4a74-9dd0-6102427a57d3/volumes"
Nov 03 22:19:31.136944 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:31.136929    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea80619-a418-4586-980e-a18da13c2b35" path="/var/lib/kubelet/pods/eea80619-a418-4586-980e-a18da13c2b35/volumes"
Nov 03 22:19:32.349471 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.349435    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"]
Nov 03 22:19:32.350105 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.350083    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" containerName="revoke-keys"
Nov 03 22:19:32.350201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.350106    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" containerName="revoke-keys"
Nov 03 22:19:32.350276 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.350263    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" containerName="revoke-keys"
Nov 03 22:19:32.353402 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.353382    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.429955 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.429922    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.430129 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.430043    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5wf\" (UniqueName: \"kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.430129 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.430092    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.456647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.456607    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:19:32.456892 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.456668    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:19:32.461604 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.461566    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"]
Nov 03 22:19:32.531591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.531558    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5wf\" (UniqueName: \"kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.531591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.531599    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.531854 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.531695    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.532439 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.532417    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.534252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.534232    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.588199 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.588161    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5wf\" (UniqueName: \"kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:32.666691 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:32.666606    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:33.041536 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:33.041513    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"]
Nov 03 22:19:33.043915 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:19:33.043888    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f621b48_e235_4104_b5c7_a246cb73cfd3.slice/crio-5f82cd3fad3e03f222f05dd0b6b1e0291439ac46d926943ec73dea6d45f06672 WatchSource:0}: Error finding container 5f82cd3fad3e03f222f05dd0b6b1e0291439ac46d926943ec73dea6d45f06672: Status 404 returned error can't find the container with id 5f82cd3fad3e03f222f05dd0b6b1e0291439ac46d926943ec73dea6d45f06672
Nov 03 22:19:33.201520 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:33.201482    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd" event={"ID":"3f621b48-e235-4104-b5c7-a246cb73cfd3","Type":"ContainerStarted","Data":"5f82cd3fad3e03f222f05dd0b6b1e0291439ac46d926943ec73dea6d45f06672"}
Nov 03 22:19:34.208840 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:34.208802    5244 generic.go:356] "Generic (PLEG): container finished" podID="3f621b48-e235-4104-b5c7-a246cb73cfd3" containerID="2ad2faae967f8566eeb9c12fbbbfe21378eb8aa777bed435983f350d32e8c81d" exitCode=0
Nov 03 22:19:34.209309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:34.208891    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd" event={"ID":"3f621b48-e235-4104-b5c7-a246cb73cfd3","Type":"ContainerDied","Data":"2ad2faae967f8566eeb9c12fbbbfe21378eb8aa777bed435983f350d32e8c81d"}
Nov 03 22:19:35.218088 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.218041    5244 generic.go:356] "Generic (PLEG): container finished" podID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerID="4bb9fb7e68faaf0af2257e9b57fa56701b8a123d82cbaf7111f15dc3466a6d27" exitCode=0
Nov 03 22:19:35.218873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.218076    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" event={"ID":"21e008d8-348f-4b8a-9f7a-23066154dad2","Type":"ContainerDied","Data":"4bb9fb7e68faaf0af2257e9b57fa56701b8a123d82cbaf7111f15dc3466a6d27"}
Nov 03 22:19:35.218873 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.218165    5244 scope.go:122] "RemoveContainer" containerID="7f5afe69b2abc43ba5a9cc426c4f6583af90dc9905895091a1d486a3d84fed72"
Nov 03 22:19:35.278363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.278278    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:19:35.367402 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367377    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367585 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367423    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367585 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367556    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367595    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367619    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367645    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367671    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367736    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367739    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:35.367910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367798    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn\") pod \"21e008d8-348f-4b8a-9f7a-23066154dad2\" (UID: \"21e008d8-348f-4b8a-9f7a-23066154dad2\") "
Nov 03 22:19:35.367910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367865    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:35.368115 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.367933    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:35.368115 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368020    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:35.368524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368278    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.368524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368309    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.368524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368345    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e008d8-348f-4b8a-9f7a-23066154dad2-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.368524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368361    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.368524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368379    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:35.368524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.368483    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:35.370006 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.369985    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:35.370264 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.370249    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "kube-api-access-hmxgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:35.370503 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.370487    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:19:35.370585 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.370511    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo" pod "21e008d8-348f-4b8a-9f7a-23066154dad2" (UID: "21e008d8-348f-4b8a-9f7a-23066154dad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:19:35.469512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.469466    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf\" (UniqueName: \"kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf\") pod \"3f621b48-e235-4104-b5c7-a246cb73cfd3\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") "
Nov 03 22:19:35.469512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.469519    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca\") pod \"3f621b48-e235-4104-b5c7-a246cb73cfd3\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") "
Nov 03 22:19:35.469803 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.469621    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca\") pod \"3f621b48-e235-4104-b5c7-a246cb73cfd3\" (UID: \"3f621b48-e235-4104-b5c7-a246cb73cfd3\") "
Nov 03 22:19:35.470096 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.470067    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca" pod "3f621b48-e235-4104-b5c7-a246cb73cfd3" (UID: "3f621b48-e235-4104-b5c7-a246cb73cfd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:35.470269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.470081    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.470269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.470134    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.470269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.470146    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.470269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.470160    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxgn\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-kube-api-access-hmxgn\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.470269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.470170    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e008d8-348f-4b8a-9f7a-23066154dad2-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.471832 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.471807    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca" pod "3f621b48-e235-4104-b5c7-a246cb73cfd3" (UID: "3f621b48-e235-4104-b5c7-a246cb73cfd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:35.471832 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.471827    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf" pod "3f621b48-e235-4104-b5c7-a246cb73cfd3" (UID: "3f621b48-e235-4104-b5c7-a246cb73cfd3"). InnerVolumeSpecName "kube-api-access-rk5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:35.571444 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.571360    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.571444 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.571387    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk5wf\" (UniqueName: \"kubernetes.io/projected/3f621b48-e235-4104-b5c7-a246cb73cfd3-kube-api-access-rk5wf\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:35.571444 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:35.571399    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f621b48-e235-4104-b5c7-a246cb73cfd3-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:36.232102 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.227500    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd" event={"ID":"3f621b48-e235-4104-b5c7-a246cb73cfd3","Type":"ContainerDied","Data":"5f82cd3fad3e03f222f05dd0b6b1e0291439ac46d926943ec73dea6d45f06672"}
Nov 03 22:19:36.232102 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.227568    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f82cd3fad3e03f222f05dd0b6b1e0291439ac46d926943ec73dea6d45f06672"
Nov 03 22:19:36.232102 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.227652    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"
Nov 03 22:19:36.258558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.257241    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x" event={"ID":"21e008d8-348f-4b8a-9f7a-23066154dad2","Type":"ContainerDied","Data":"6aca8c4751e871d33dde3f8a0060fee91f0a52d0865756709a4c72785e781f02"}
Nov 03 22:19:36.258558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.257314    5244 scope.go:122] "RemoveContainer" containerID="4bb9fb7e68faaf0af2257e9b57fa56701b8a123d82cbaf7111f15dc3466a6d27"
Nov 03 22:19:36.258558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.257494    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"
Nov 03 22:19:36.420345 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.420300    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"]
Nov 03 22:19:36.469830 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.469785    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-288188-openshift-default-69d944b649-w4b8x"]
Nov 03 22:19:36.989812 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.989774    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"]
Nov 03 22:19:36.990272 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.990259    5244 cpu_manager.go:414] "containerMap: removing container" podUID="3f621b48-e235-4104-b5c7-a246cb73cfd3" containerName="revoke-keys"
Nov 03 22:19:36.990322 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.990274    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f621b48-e235-4104-b5c7-a246cb73cfd3" containerName="revoke-keys"
Nov 03 22:19:36.990322 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.990288    5244 cpu_manager.go:414] "containerMap: removing container" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy"
Nov 03 22:19:36.990322 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.990293    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy"
Nov 03 22:19:36.990438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.990400    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy"
Nov 03 22:19:36.990438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.990413    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f621b48-e235-4104-b5c7-a246cb73cfd3" containerName="revoke-keys"
Nov 03 22:19:36.995405 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:36.995386    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.047990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.047956    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:19:37.048145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.048034    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:19:37.049973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.049944    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"]
Nov 03 22:19:37.089730 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.089699    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.089730 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.089733    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.089940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.089770    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mgx\" (UniqueName: \"kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.136317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.136284    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" path="/var/lib/kubelet/pods/21e008d8-348f-4b8a-9f7a-23066154dad2/volumes"
Nov 03 22:19:37.190814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.190776    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.190814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.190812    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.191086 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.190847    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mgx\" (UniqueName: \"kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.191534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.191507    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.193544 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.193520    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.248149 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.248077    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mgx\" (UniqueName: \"kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.309119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.309083    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:37.705848 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:37.705819    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"]
Nov 03 22:19:37.707991 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:19:37.707963    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b96d05_d64a_4b3c_af1d_047dc1e2e58e.slice/crio-126c842f50cbf3b1105e8eb8a27b44d0b06febbacc7fb3418bae14dcdaa9faec WatchSource:0}: Error finding container 126c842f50cbf3b1105e8eb8a27b44d0b06febbacc7fb3418bae14dcdaa9faec: Status 404 returned error can't find the container with id 126c842f50cbf3b1105e8eb8a27b44d0b06febbacc7fb3418bae14dcdaa9faec
Nov 03 22:19:38.271091 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:38.271052    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr" event={"ID":"09b96d05-d64a-4b3c-af1d-047dc1e2e58e","Type":"ContainerStarted","Data":"13b5bb04bd9d84b8110f322191e92b12b8e72616da77126c6ce2e4820775a161"}
Nov 03 22:19:38.271539 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:38.271119    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr" event={"ID":"09b96d05-d64a-4b3c-af1d-047dc1e2e58e","Type":"ContainerStarted","Data":"126c842f50cbf3b1105e8eb8a27b44d0b06febbacc7fb3418bae14dcdaa9faec"}
Nov 03 22:19:38.384595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:38.384495    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr" podStartSLOduration=2.384479123 podStartE2EDuration="2.384479123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:38.382906527 +0000 UTC m=+3849.366903087" watchObservedRunningTime="2026-11-03 22:19:38.384479123 +0000 UTC m=+3849.368475681"
Nov 03 22:19:39.278720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:39.278682    5244 generic.go:356] "Generic (PLEG): container finished" podID="09b96d05-d64a-4b3c-af1d-047dc1e2e58e" containerID="13b5bb04bd9d84b8110f322191e92b12b8e72616da77126c6ce2e4820775a161" exitCode=0
Nov 03 22:19:39.279265 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:39.278776    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr" event={"ID":"09b96d05-d64a-4b3c-af1d-047dc1e2e58e","Type":"ContainerDied","Data":"13b5bb04bd9d84b8110f322191e92b12b8e72616da77126c6ce2e4820775a161"}
Nov 03 22:19:39.869502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:39.869458    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:19:40.421241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.421213    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:40.538262 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.538161    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca\") pod \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") "
Nov 03 22:19:40.538478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.538305    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca\") pod \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") "
Nov 03 22:19:40.538478 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.538421    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx\" (UniqueName: \"kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx\") pod \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\" (UID: \"09b96d05-d64a-4b3c-af1d-047dc1e2e58e\") "
Nov 03 22:19:40.538773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.538744    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca" pod "09b96d05-d64a-4b3c-af1d-047dc1e2e58e" (UID: "09b96d05-d64a-4b3c-af1d-047dc1e2e58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:19:40.538913 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.538833    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:40.541263 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.541233    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca" pod "09b96d05-d64a-4b3c-af1d-047dc1e2e58e" (UID: "09b96d05-d64a-4b3c-af1d-047dc1e2e58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:19:40.541263 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.541254    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx" pod "09b96d05-d64a-4b3c-af1d-047dc1e2e58e" (UID: "09b96d05-d64a-4b3c-af1d-047dc1e2e58e"). InnerVolumeSpecName "kube-api-access-r5mgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:19:40.639563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.639527    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mgx\" (UniqueName: \"kubernetes.io/projected/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-kube-api-access-r5mgx\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:40.639563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:40.639555    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b96d05-d64a-4b3c-af1d-047dc1e2e58e-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:19:41.045589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:41.045548    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"]
Nov 03 22:19:41.047824 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:41.047798    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-131c669b-llstx"]
Nov 03 22:19:41.138534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:41.138501    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8676dea0-0e70-4c0e-8c2f-83063202f562" path="/var/lib/kubelet/pods/8676dea0-0e70-4c0e-8c2f-83063202f562/volumes"
Nov 03 22:19:41.291088 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:41.291058    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"
Nov 03 22:19:41.291269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:41.291101    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr" event={"ID":"09b96d05-d64a-4b3c-af1d-047dc1e2e58e","Type":"ContainerDied","Data":"126c842f50cbf3b1105e8eb8a27b44d0b06febbacc7fb3418bae14dcdaa9faec"}
Nov 03 22:19:41.291269 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:19:41.291150    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126c842f50cbf3b1105e8eb8a27b44d0b06febbacc7fb3418bae14dcdaa9faec"
Nov 03 22:19:52.436921 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:19:52.436883    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:20:12.025141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:12.019084    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:20:15.478027 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:15.477987    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"]
Nov 03 22:20:15.527258 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:15.527221    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d7b12e-6f2rl"]
Nov 03 22:20:17.135809 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:17.135772    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06552027-4c4b-4a63-8f07-f2f72c986463" path="/var/lib/kubelet/pods/06552027-4c4b-4a63-8f07-f2f72c986463/volumes"
Nov 03 22:20:22.174360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:22.174303    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"]
Nov 03 22:20:22.174928 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:22.174610    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy" containerID="cri-o://9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61" gracePeriod=30
Nov 03 22:20:22.441000 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:22.440957    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:20:24.318951 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.318914    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"]
Nov 03 22:20:24.319401 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.319386    5244 cpu_manager.go:414] "containerMap: removing container" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy"
Nov 03 22:20:24.319456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.319401    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy"
Nov 03 22:20:24.319456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.319432    5244 cpu_manager.go:414] "containerMap: removing container" podUID="09b96d05-d64a-4b3c-af1d-047dc1e2e58e" containerName="revoke-keys"
Nov 03 22:20:24.319456 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.319437    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b96d05-d64a-4b3c-af1d-047dc1e2e58e" containerName="revoke-keys"
Nov 03 22:20:24.319558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.319514    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b96d05-d64a-4b3c-af1d-047dc1e2e58e" containerName="revoke-keys"
Nov 03 22:20:24.319558 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.319525    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e008d8-348f-4b8a-9f7a-23066154dad2" containerName="istio-proxy"
Nov 03 22:20:24.323991 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.323975    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.379308 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.379264    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"]
Nov 03 22:20:24.425795 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.425760    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:20:24.425970 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.425843    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:20:24.447877 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.447837    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.448094 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.447883    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.448094 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.447985    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl96f\" (UniqueName: \"kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.549002 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.548963    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.549002 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.549001    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.549301 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.549034    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl96f\" (UniqueName: \"kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.549747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.549721    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.551593 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.551570    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.613172 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.613088    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl96f\" (UniqueName: \"kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:24.637682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:24.637656    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:25.017715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:25.017645    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"]
Nov 03 22:20:25.020489 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:20:25.020451    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cac9d1_1540_4da4_acc6_55b5124b98a7.slice/crio-4b0e4634e5a6544ee5d0a4597d1dfed4992936e618a9335cd8ae3069f5eb52e6 WatchSource:0}: Error finding container 4b0e4634e5a6544ee5d0a4597d1dfed4992936e618a9335cd8ae3069f5eb52e6: Status 404 returned error can't find the container with id 4b0e4634e5a6544ee5d0a4597d1dfed4992936e618a9335cd8ae3069f5eb52e6
Nov 03 22:20:25.568375 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:25.568313    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58" event={"ID":"51cac9d1-1540-4da4-acc6-55b5124b98a7","Type":"ContainerStarted","Data":"b5d95c2c07bf59e8cb273033bf88515b885c58cd041aa665d039f5587dcf0c2f"}
Nov 03 22:20:25.568786 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:25.568397    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58" event={"ID":"51cac9d1-1540-4da4-acc6-55b5124b98a7","Type":"ContainerStarted","Data":"4b0e4634e5a6544ee5d0a4597d1dfed4992936e618a9335cd8ae3069f5eb52e6"}
Nov 03 22:20:25.680555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:25.680420    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58" podStartSLOduration=1.680399692 podStartE2EDuration="1.680399692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:25.679938853 +0000 UTC m=+3896.663935412" watchObservedRunningTime="2026-11-03 22:20:25.680399692 +0000 UTC m=+3896.664396253"
Nov 03 22:20:26.575871 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:26.575837    5244 generic.go:356] "Generic (PLEG): container finished" podID="51cac9d1-1540-4da4-acc6-55b5124b98a7" containerID="b5d95c2c07bf59e8cb273033bf88515b885c58cd041aa665d039f5587dcf0c2f" exitCode=0
Nov 03 22:20:26.576292 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:26.575916    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58" event={"ID":"51cac9d1-1540-4da4-acc6-55b5124b98a7","Type":"ContainerDied","Data":"b5d95c2c07bf59e8cb273033bf88515b885c58cd041aa665d039f5587dcf0c2f"}
Nov 03 22:20:27.439393 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.439363    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:20:27.583459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583369    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.583459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583413    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.583459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583432    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583473    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583538    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583580    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583605    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583680    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583737    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert\") pod \"b3be1e24-014c-4657-b8a7-2a277e135696\" (UID: \"b3be1e24-014c-4657-b8a7-2a277e135696\") "
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583768    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.583812    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:27.584053 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584006    5244 generic.go:356] "Generic (PLEG): container finished" podID="b3be1e24-014c-4657-b8a7-2a277e135696" containerID="9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61" exitCode=0
Nov 03 22:20:27.584545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584410    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"
Nov 03 22:20:27.584545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584421    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.584545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584443    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.584545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584481    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:27.584545 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584530    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" event={"ID":"b3be1e24-014c-4657-b8a7-2a277e135696","Type":"ContainerDied","Data":"9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61"}
Nov 03 22:20:27.584790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584602    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n" event={"ID":"b3be1e24-014c-4657-b8a7-2a277e135696","Type":"ContainerDied","Data":"b3e7706f1b29a6b4ed389c21b165f9c65c76b3d32c9063d69c419089891db717"}
Nov 03 22:20:27.584790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584630    5244 scope.go:122] "RemoveContainer" containerID="9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61"
Nov 03 22:20:27.585024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584933    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:27.585024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.584971    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:27.586296 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.586259    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:20:27.587196 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.587170    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:27.587374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.587314    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:20:27.588466 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.588442    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk" pod "b3be1e24-014c-4657-b8a7-2a277e135696" (UID: "b3be1e24-014c-4657-b8a7-2a277e135696"). InnerVolumeSpecName "kube-api-access-jglfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:27.613983 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.613952    5244 scope.go:122] "RemoveContainer" containerID="f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"
Nov 03 22:20:27.624381 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.624352    5244 scope.go:122] "RemoveContainer" containerID="9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61"
Nov 03 22:20:27.624734 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:27.624712    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61\": container with ID starting with 9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61 not found: ID does not exist" containerID="9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61"
Nov 03 22:20:27.624823 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.624744    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61"} err="failed to get container status \"9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61\": rpc error: code = NotFound desc = could not find container \"9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61\": container with ID starting with 9b31fbca98db7dea513dff6e7e32ba323ef9a0d10f2ff9c6e9d61a6715fb2c61 not found: ID does not exist"
Nov 03 22:20:27.624823 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.624764    5244 scope.go:122] "RemoveContainer" containerID="f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"
Nov 03 22:20:27.625024 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:27.625009    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51\": container with ID starting with f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51 not found: ID does not exist" containerID="f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"
Nov 03 22:20:27.625072 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.625027    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51"} err="failed to get container status \"f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51\": rpc error: code = NotFound desc = could not find container \"f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51\": container with ID starting with f76a9aa6cedeb2195648f2ccd4ca5c13d594e4b4a8022b136561169f38b94d51 not found: ID does not exist"
Nov 03 22:20:27.685596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685558    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jglfk\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-kube-api-access-jglfk\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.685763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685600    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.685763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685621    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.685763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685635    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3be1e24-014c-4657-b8a7-2a277e135696-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.685763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685650    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3be1e24-014c-4657-b8a7-2a277e135696-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.685763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685665    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3be1e24-014c-4657-b8a7-2a277e135696-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.685763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.685679    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3be1e24-014c-4657-b8a7-2a277e135696-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.731105 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.731076    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:27.887557 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.887475    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca\") pod \"51cac9d1-1540-4da4-acc6-55b5124b98a7\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") "
Nov 03 22:20:27.887732 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.887593    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f\" (UniqueName: \"kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f\") pod \"51cac9d1-1540-4da4-acc6-55b5124b98a7\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") "
Nov 03 22:20:27.887732 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.887708    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca\") pod \"51cac9d1-1540-4da4-acc6-55b5124b98a7\" (UID: \"51cac9d1-1540-4da4-acc6-55b5124b98a7\") "
Nov 03 22:20:27.888155 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.888112    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca" pod "51cac9d1-1540-4da4-acc6-55b5124b98a7" (UID: "51cac9d1-1540-4da4-acc6-55b5124b98a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:27.889836 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.889813    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca" pod "51cac9d1-1540-4da4-acc6-55b5124b98a7" (UID: "51cac9d1-1540-4da4-acc6-55b5124b98a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:27.889924 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.889861    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f" pod "51cac9d1-1540-4da4-acc6-55b5124b98a7" (UID: "51cac9d1-1540-4da4-acc6-55b5124b98a7"). InnerVolumeSpecName "kube-api-access-xl96f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:27.989369 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.989320    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl96f\" (UniqueName: \"kubernetes.io/projected/51cac9d1-1540-4da4-acc6-55b5124b98a7-kube-api-access-xl96f\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.989369 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.989365    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:27.989369 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:27.989376    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cac9d1-1540-4da4-acc6-55b5124b98a7-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:28.004679 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:28.004648    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"]
Nov 03 22:20:28.008068 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:28.008041    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d7b12e-openshift-default-5b65698697-q2l5n"]
Nov 03 22:20:28.591645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:28.591559    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"
Nov 03 22:20:28.591645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:28.591585    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58" event={"ID":"51cac9d1-1540-4da4-acc6-55b5124b98a7","Type":"ContainerDied","Data":"4b0e4634e5a6544ee5d0a4597d1dfed4992936e618a9335cd8ae3069f5eb52e6"}
Nov 03 22:20:28.591645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:28.591640    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0e4634e5a6544ee5d0a4597d1dfed4992936e618a9335cd8ae3069f5eb52e6"
Nov 03 22:20:29.137579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:29.137541    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" path="/var/lib/kubelet/pods/b3be1e24-014c-4657-b8a7-2a277e135696/volumes"
Nov 03 22:20:30.796429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:30.796397    5244 scope.go:122] "RemoveContainer" containerID="cd0dc8ef49fe3db10a3c041143f7831a98d9be45a7322389bcb8fe07b6781a82"
Nov 03 22:20:30.805991 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:30.805970    5244 scope.go:122] "RemoveContainer" containerID="ec003b726ca3cf7c7fb775d151898e38e3b5d44a7321bd737dd9b1fe5754745c"
Nov 03 22:20:35.885308 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:35.885275    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"]
Nov 03 22:20:35.992384 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:35.992323    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"]
Nov 03 22:20:40.886267 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:40.886189    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm" podUID="c6da5f32-0288-4e77-8599-deaaab3ca3f6" containerName="payload-processing" containerID="cri-o://1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615" gracePeriod=25
Nov 03 22:20:40.992790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:40.992714    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j" podUID="49c4b706-d80c-41e7-81b0-e51025177750" containerName="payload-pre-processing" containerID="cri-o://3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81" gracePeriod=25
Nov 03 22:20:41.152623 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.152551    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:20:41.226860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.226809    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume\") pod \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") "
Nov 03 22:20:41.227049 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.226980    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9\" (UniqueName: \"kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9\") pod \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\" (UID: \"c6da5f32-0288-4e77-8599-deaaab3ca3f6\") "
Nov 03 22:20:41.227210 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.227182    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume" pod "c6da5f32-0288-4e77-8599-deaaab3ca3f6" (UID: "c6da5f32-0288-4e77-8599-deaaab3ca3f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:41.227401 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.227384    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da5f32-0288-4e77-8599-deaaab3ca3f6-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:41.229278 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.229251    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9" pod "c6da5f32-0288-4e77-8599-deaaab3ca3f6" (UID: "c6da5f32-0288-4e77-8599-deaaab3ca3f6"). InnerVolumeSpecName "kube-api-access-8cpx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:41.246816 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.246791    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:20:41.328458 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.328421    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume\") pod \"49c4b706-d80c-41e7-81b0-e51025177750\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") "
Nov 03 22:20:41.328657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.328568    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4\" (UniqueName: \"kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4\") pod \"49c4b706-d80c-41e7-81b0-e51025177750\" (UID: \"49c4b706-d80c-41e7-81b0-e51025177750\") "
Nov 03 22:20:41.328797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.328783    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cpx9\" (UniqueName: \"kubernetes.io/projected/c6da5f32-0288-4e77-8599-deaaab3ca3f6-kube-api-access-8cpx9\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:41.328867 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.328796    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume" pod "49c4b706-d80c-41e7-81b0-e51025177750" (UID: "49c4b706-d80c-41e7-81b0-e51025177750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:41.330998 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.330971    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4" pod "49c4b706-d80c-41e7-81b0-e51025177750" (UID: "49c4b706-d80c-41e7-81b0-e51025177750"). InnerVolumeSpecName "kube-api-access-9cst4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:41.430342 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.430247    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cst4\" (UniqueName: \"kubernetes.io/projected/49c4b706-d80c-41e7-81b0-e51025177750-kube-api-access-9cst4\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:41.430342 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.430276    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c4b706-d80c-41e7-81b0-e51025177750-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:41.676013 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.675975    5244 generic.go:356] "Generic (PLEG): container finished" podID="49c4b706-d80c-41e7-81b0-e51025177750" containerID="3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81" exitCode=0
Nov 03 22:20:41.676222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.676032    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"
Nov 03 22:20:41.676222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.676056    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j" event={"ID":"49c4b706-d80c-41e7-81b0-e51025177750","Type":"ContainerDied","Data":"3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81"}
Nov 03 22:20:41.676222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.676138    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j" event={"ID":"49c4b706-d80c-41e7-81b0-e51025177750","Type":"ContainerDied","Data":"67ea267a5ff02e9af306887daf0f584431259c6e47e209d9c7bb6d0ebc7d376d"}
Nov 03 22:20:41.676222 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.676161    5244 scope.go:122] "RemoveContainer" containerID="3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81"
Nov 03 22:20:41.677731 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.677704    5244 generic.go:356] "Generic (PLEG): container finished" podID="c6da5f32-0288-4e77-8599-deaaab3ca3f6" containerID="1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615" exitCode=0
Nov 03 22:20:41.677860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.677769    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"
Nov 03 22:20:41.677860 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.677821    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm" event={"ID":"c6da5f32-0288-4e77-8599-deaaab3ca3f6","Type":"ContainerDied","Data":"1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615"}
Nov 03 22:20:41.677946 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.677860    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm" event={"ID":"c6da5f32-0288-4e77-8599-deaaab3ca3f6","Type":"ContainerDied","Data":"8bfa55c572154440f9483e16f79beb94aaf2d96746e24e4814364fc107a351e0"}
Nov 03 22:20:41.686521 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.686498    5244 scope.go:122] "RemoveContainer" containerID="3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81"
Nov 03 22:20:41.686856 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:41.686832    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81\": container with ID starting with 3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81 not found: ID does not exist" containerID="3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81"
Nov 03 22:20:41.686964 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.686865    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81"} err="failed to get container status \"3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81\": rpc error: code = NotFound desc = could not find container \"3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81\": container with ID starting with 3fccd176446616ae6a3aefc176171e334370bfb8b0fb9d168f0972d89233fe81 not found: ID does not exist"
Nov 03 22:20:41.686964 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.686885    5244 scope.go:122] "RemoveContainer" containerID="1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615"
Nov 03 22:20:41.697160 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.697134    5244 scope.go:122] "RemoveContainer" containerID="1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615"
Nov 03 22:20:41.697534 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:41.697510    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615\": container with ID starting with 1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615 not found: ID does not exist" containerID="1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615"
Nov 03 22:20:41.697616 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.697541    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615"} err="failed to get container status \"1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615\": rpc error: code = NotFound desc = could not find container \"1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615\": container with ID starting with 1b2d1fcb05a501c73f2932a96682aff8569dd504dcc5dbec826fd8030396d615 not found: ID does not exist"
Nov 03 22:20:41.838383 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.838349    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"]
Nov 03 22:20:41.888111 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.888069    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a49a32-9889555bc-m7h8j"]
Nov 03 22:20:41.949851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.949747    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"]
Nov 03 22:20:41.951857 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:41.951837    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a49a32-7c89f5675b-dp2nm"]
Nov 03 22:20:43.135470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:43.135439    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c4b706-d80c-41e7-81b0-e51025177750" path="/var/lib/kubelet/pods/49c4b706-d80c-41e7-81b0-e51025177750/volumes"
Nov 03 22:20:43.135937 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:43.135812    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6da5f32-0288-4e77-8599-deaaab3ca3f6" path="/var/lib/kubelet/pods/c6da5f32-0288-4e77-8599-deaaab3ca3f6/volumes"
Nov 03 22:20:45.282033 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:45.281992    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"]
Nov 03 22:20:45.330945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:45.330914    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f3e839-mw9x4"]
Nov 03 22:20:47.136004 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:47.135973    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b0b9b2-531c-4021-b8cc-bd2cf1866a51" path="/var/lib/kubelet/pods/f8b0b9b2-531c-4021-b8cc-bd2cf1866a51/volumes"
Nov 03 22:20:52.437147 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:52.437103    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:20:54.070298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.070258    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"]
Nov 03 22:20:54.070962 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.070940    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c6da5f32-0288-4e77-8599-deaaab3ca3f6" containerName="payload-processing"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.070963    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6da5f32-0288-4e77-8599-deaaab3ca3f6" containerName="payload-processing"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.070977    5244 cpu_manager.go:414] "containerMap: removing container" podUID="51cac9d1-1540-4da4-acc6-55b5124b98a7" containerName="revoke-keys"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.070984    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cac9d1-1540-4da4-acc6-55b5124b98a7" containerName="revoke-keys"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071015    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071023    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071035    5244 cpu_manager.go:414] "containerMap: removing container" podUID="49c4b706-d80c-41e7-81b0-e51025177750" containerName="payload-pre-processing"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071043    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c4b706-d80c-41e7-81b0-e51025177750" containerName="payload-pre-processing"
Nov 03 22:20:54.071059 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071056    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy"
Nov 03 22:20:54.071502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071063    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy"
Nov 03 22:20:54.071502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071192    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6da5f32-0288-4e77-8599-deaaab3ca3f6" containerName="payload-processing"
Nov 03 22:20:54.071502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071203    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cac9d1-1540-4da4-acc6-55b5124b98a7" containerName="revoke-keys"
Nov 03 22:20:54.071502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071216    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c4b706-d80c-41e7-81b0-e51025177750" containerName="payload-pre-processing"
Nov 03 22:20:54.071502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071225    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy"
Nov 03 22:20:54.071502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.071235    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3be1e24-014c-4657-b8a7-2a277e135696" containerName="istio-proxy"
Nov 03 22:20:54.074715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.074695    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.128552 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.128510    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:20:54.177214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.177171    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.177420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.177379    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chb2q\" (UniqueName: \"kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.177493 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.177458    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.178166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.178145    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:20:54.180107 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.180089    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"]
Nov 03 22:20:54.278798 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.278764    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chb2q\" (UniqueName: \"kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.279008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.278832    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.279008 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.278872    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.279617 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.279596    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.281516 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.281490    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.335657 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.335584    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chb2q\" (UniqueName: \"kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.387644 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.387609    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:54.762912 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:54.762891    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"]
Nov 03 22:20:54.765615 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:20:54.765583    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9d6338_5031_4546_854a_de49316b03b8.slice/crio-f6e6c960dc57a3a4f47eb8d9467fc68bd1f48f17c98bcc484e1697b1ec813095 WatchSource:0}: Error finding container f6e6c960dc57a3a4f47eb8d9467fc68bd1f48f17c98bcc484e1697b1ec813095: Status 404 returned error can't find the container with id f6e6c960dc57a3a4f47eb8d9467fc68bd1f48f17c98bcc484e1697b1ec813095
Nov 03 22:20:55.764663 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:55.764627    5244 generic.go:356] "Generic (PLEG): container finished" podID="8a9d6338-5031-4546-854a-de49316b03b8" containerID="20889961bd8ba19d731b96e8bedc53aa4c1c6ebebd2d4f11a431d5ecf925c364" exitCode=0
Nov 03 22:20:55.765077 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:55.764717    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l" event={"ID":"8a9d6338-5031-4546-854a-de49316b03b8","Type":"ContainerDied","Data":"20889961bd8ba19d731b96e8bedc53aa4c1c6ebebd2d4f11a431d5ecf925c364"}
Nov 03 22:20:55.765077 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:55.764767    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l" event={"ID":"8a9d6338-5031-4546-854a-de49316b03b8","Type":"ContainerStarted","Data":"f6e6c960dc57a3a4f47eb8d9467fc68bd1f48f17c98bcc484e1697b1ec813095"}
Nov 03 22:20:56.923028 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:56.923004    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:57.009234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.009198    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca\") pod \"8a9d6338-5031-4546-854a-de49316b03b8\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") "
Nov 03 22:20:57.009467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.009251    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q\" (UniqueName: \"kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q\") pod \"8a9d6338-5031-4546-854a-de49316b03b8\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") "
Nov 03 22:20:57.009467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.009309    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca\") pod \"8a9d6338-5031-4546-854a-de49316b03b8\" (UID: \"8a9d6338-5031-4546-854a-de49316b03b8\") "
Nov 03 22:20:57.009765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.009735    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca" pod "8a9d6338-5031-4546-854a-de49316b03b8" (UID: "8a9d6338-5031-4546-854a-de49316b03b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:57.011907 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.011876    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q" pod "8a9d6338-5031-4546-854a-de49316b03b8" (UID: "8a9d6338-5031-4546-854a-de49316b03b8"). InnerVolumeSpecName "kube-api-access-chb2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:57.012050 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.012027    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca" pod "8a9d6338-5031-4546-854a-de49316b03b8" (UID: "8a9d6338-5031-4546-854a-de49316b03b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:57.111021 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.110937    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chb2q\" (UniqueName: \"kubernetes.io/projected/8a9d6338-5031-4546-854a-de49316b03b8-kube-api-access-chb2q\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.111021 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.110966    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.111021 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.110977    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a9d6338-5031-4546-854a-de49316b03b8-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:57.777920 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.777891    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"
Nov 03 22:20:57.777920 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.777918    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l" event={"ID":"8a9d6338-5031-4546-854a-de49316b03b8","Type":"ContainerDied","Data":"f6e6c960dc57a3a4f47eb8d9467fc68bd1f48f17c98bcc484e1697b1ec813095"}
Nov 03 22:20:57.778151 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:57.777983    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6e6c960dc57a3a4f47eb8d9467fc68bd1f48f17c98bcc484e1697b1ec813095"
Nov 03 22:20:59.337058 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.337018    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"]
Nov 03 22:20:59.337566 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.337384    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" podUID="2dfbf5b6-d4b6-4216-b674-336324640a2d" containerName="maas-api" containerID="cri-o://f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77" gracePeriod=30
Nov 03 22:20:59.654009 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.653982    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:20:59.662089 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.662059    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\") pod \"2dfbf5b6-d4b6-4216-b674-336324640a2d\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") "
Nov 03 22:20:59.662256 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.662103    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp\" (UniqueName: \"kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp\") pod \"2dfbf5b6-d4b6-4216-b674-336324640a2d\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") "
Nov 03 22:20:59.662256 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.662142    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs\") pod \"2dfbf5b6-d4b6-4216-b674-336324640a2d\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") "
Nov 03 22:20:59.662256 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.662250    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca\") pod \"2dfbf5b6-d4b6-4216-b674-336324640a2d\" (UID: \"2dfbf5b6-d4b6-4216-b674-336324640a2d\") "
Nov 03 22:20:59.662775 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.662750    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca" pod "2dfbf5b6-d4b6-4216-b674-336324640a2d" (UID: "2dfbf5b6-d4b6-4216-b674-336324640a2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:20:59.664918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.664892    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs" pod "2dfbf5b6-d4b6-4216-b674-336324640a2d" (UID: "2dfbf5b6-d4b6-4216-b674-336324640a2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:59.665024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.664934    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls" pod "2dfbf5b6-d4b6-4216-b674-336324640a2d" (UID: "2dfbf5b6-d4b6-4216-b674-336324640a2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:20:59.665024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.664939    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp" pod "2dfbf5b6-d4b6-4216-b674-336324640a2d" (UID: "2dfbf5b6-d4b6-4216-b674-336324640a2d"). InnerVolumeSpecName "kube-api-access-lqmtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:20:59.763640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.763591    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:59.763640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.763637    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfbf5b6-d4b6-4216-b674-336324640a2d-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:59.763917 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.763655    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfbf5b6-d4b6-4216-b674-336324640a2d-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:59.763917 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.763669    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmtp\" (UniqueName: \"kubernetes.io/projected/2dfbf5b6-d4b6-4216-b674-336324640a2d-kube-api-access-lqmtp\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:20:59.792214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.792179    5244 generic.go:356] "Generic (PLEG): container finished" podID="2dfbf5b6-d4b6-4216-b674-336324640a2d" containerID="f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77" exitCode=0
Nov 03 22:20:59.792420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.792237    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" event={"ID":"2dfbf5b6-d4b6-4216-b674-336324640a2d","Type":"ContainerDied","Data":"f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77"}
Nov 03 22:20:59.792420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.792245    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"
Nov 03 22:20:59.792420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.792287    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j" event={"ID":"2dfbf5b6-d4b6-4216-b674-336324640a2d","Type":"ContainerDied","Data":"342f47c558df98106e7de93653cc4122d6be98c5786b4232720a16821bc59611"}
Nov 03 22:20:59.792420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.792318    5244 scope.go:122] "RemoveContainer" containerID="f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77"
Nov 03 22:20:59.802685 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.802663    5244 scope.go:122] "RemoveContainer" containerID="f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77"
Nov 03 22:20:59.803174 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:20:59.803147    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77\": container with ID starting with f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77 not found: ID does not exist" containerID="f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77"
Nov 03 22:20:59.803287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.803184    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77"} err="failed to get container status \"f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77\": rpc error: code = NotFound desc = could not find container \"f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77\": container with ID starting with f281ff71f761cee1c1491debbd0823d87c4ca86ff634357759fd4ccada517d77 not found: ID does not exist"
Nov 03 22:20:59.955250 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.955213    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"]
Nov 03 22:20:59.958613 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:20:59.958588    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-29dbd9-577fcbb557-llg7j"]
Nov 03 22:21:01.160799 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:01.160762    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dfbf5b6-d4b6-4216-b674-336324640a2d" path="/var/lib/kubelet/pods/2dfbf5b6-d4b6-4216-b674-336324640a2d/volumes"
Nov 03 22:21:01.526054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:01.526019    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"]
Nov 03 22:21:01.575734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:01.575698    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-29dbd9-8ldwd"]
Nov 03 22:21:02.850665 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:02.850634    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"]
Nov 03 22:21:02.892244 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:02.892201    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-534c77-wxbjr"]
Nov 03 22:21:03.135908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.135826    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b96d05-d64a-4b3c-af1d-047dc1e2e58e" path="/var/lib/kubelet/pods/09b96d05-d64a-4b3c-af1d-047dc1e2e58e/volumes"
Nov 03 22:21:03.136242 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.136228    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f621b48-e235-4104-b5c7-a246cb73cfd3" path="/var/lib/kubelet/pods/3f621b48-e235-4104-b5c7-a246cb73cfd3/volumes"
Nov 03 22:21:03.855246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855210    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"]
Nov 03 22:21:03.855686 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855664    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8a9d6338-5031-4546-854a-de49316b03b8" containerName="revoke-keys"
Nov 03 22:21:03.855686 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855676    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9d6338-5031-4546-854a-de49316b03b8" containerName="revoke-keys"
Nov 03 22:21:03.855788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855696    5244 cpu_manager.go:414] "containerMap: removing container" podUID="2dfbf5b6-d4b6-4216-b674-336324640a2d" containerName="maas-api"
Nov 03 22:21:03.855788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855701    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dfbf5b6-d4b6-4216-b674-336324640a2d" containerName="maas-api"
Nov 03 22:21:03.855788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855772    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9d6338-5031-4546-854a-de49316b03b8" containerName="revoke-keys"
Nov 03 22:21:03.855788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.855781    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dfbf5b6-d4b6-4216-b674-336324640a2d" containerName="maas-api"
Nov 03 22:21:03.858968 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.858952    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.896799 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.896764    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.896949 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.896894    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.897030 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.897014    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h894k\" (UniqueName: \"kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.963279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.963245    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:21:03.963279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.963276    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:21:03.965367 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.965313    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"]
Nov 03 22:21:03.997956 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.997920    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h894k\" (UniqueName: \"kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.998144 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.997975    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.998144 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.998018    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:03.998769 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:03.998746    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:04.000555 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:04.000524    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:04.054286 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:04.054240    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h894k\" (UniqueName: \"kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k\") pod \"maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:04.172523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:04.172435    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:04.544909 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:04.544880    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"]
Nov 03 22:21:04.547436 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:21:04.547404    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f48daa0_fb13_4d8a_9ec9_5eb17b714143.slice/crio-cff82d4c217da82dd55d523c41fd3a4ba8c76aabeb90d288dcba399fb5fffec2 WatchSource:0}: Error finding container cff82d4c217da82dd55d523c41fd3a4ba8c76aabeb90d288dcba399fb5fffec2: Status 404 returned error can't find the container with id cff82d4c217da82dd55d523c41fd3a4ba8c76aabeb90d288dcba399fb5fffec2
Nov 03 22:21:04.845126 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:04.845098    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp" event={"ID":"8f48daa0-fb13-4d8a-9ec9-5eb17b714143","Type":"ContainerStarted","Data":"cff82d4c217da82dd55d523c41fd3a4ba8c76aabeb90d288dcba399fb5fffec2"}
Nov 03 22:21:04.906154 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:04.906122    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:21:05.855121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:05.855088    5244 generic.go:356] "Generic (PLEG): container finished" podID="8f48daa0-fb13-4d8a-9ec9-5eb17b714143" containerID="4191aabf41002b805ab6d1eb8b2abfb6498bd719cbe9f83707879e920d0e992e" exitCode=0
Nov 03 22:21:05.855319 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:05.855184    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp" event={"ID":"8f48daa0-fb13-4d8a-9ec9-5eb17b714143","Type":"ContainerDied","Data":"4191aabf41002b805ab6d1eb8b2abfb6498bd719cbe9f83707879e920d0e992e"}
Nov 03 22:21:07.025484 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.025455    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:07.138408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.137986    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca\") pod \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") "
Nov 03 22:21:07.138408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.138074    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca\") pod \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") "
Nov 03 22:21:07.138408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.138137    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k\" (UniqueName: \"kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k\") pod \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\" (UID: \"8f48daa0-fb13-4d8a-9ec9-5eb17b714143\") "
Nov 03 22:21:07.142460 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.140751    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca" pod "8f48daa0-fb13-4d8a-9ec9-5eb17b714143" (UID: "8f48daa0-fb13-4d8a-9ec9-5eb17b714143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:07.142460 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.141026    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca" pod "8f48daa0-fb13-4d8a-9ec9-5eb17b714143" (UID: "8f48daa0-fb13-4d8a-9ec9-5eb17b714143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:07.142460 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.141421    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k" pod "8f48daa0-fb13-4d8a-9ec9-5eb17b714143" (UID: "8f48daa0-fb13-4d8a-9ec9-5eb17b714143"). InnerVolumeSpecName "kube-api-access-h894k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:07.239901 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.239867    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:07.239901 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.239897    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:07.239901 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.239909    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h894k\" (UniqueName: \"kubernetes.io/projected/8f48daa0-fb13-4d8a-9ec9-5eb17b714143-kube-api-access-h894k\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:07.869512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.869482    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"
Nov 03 22:21:07.869742 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.869531    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp" event={"ID":"8f48daa0-fb13-4d8a-9ec9-5eb17b714143","Type":"ContainerDied","Data":"cff82d4c217da82dd55d523c41fd3a4ba8c76aabeb90d288dcba399fb5fffec2"}
Nov 03 22:21:07.869742 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.869590    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff82d4c217da82dd55d523c41fd3a4ba8c76aabeb90d288dcba399fb5fffec2"
Nov 03 22:21:07.927916 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.927879    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"]
Nov 03 22:21:07.928199 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:07.928172    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy" containerID="cri-o://2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e" gracePeriod=30
Nov 03 22:21:08.734286 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.734253    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"]
Nov 03 22:21:08.734701 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.734689    5244 cpu_manager.go:414] "containerMap: removing container" podUID="8f48daa0-fb13-4d8a-9ec9-5eb17b714143" containerName="revoke-keys"
Nov 03 22:21:08.734758 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.734703    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f48daa0-fb13-4d8a-9ec9-5eb17b714143" containerName="revoke-keys"
Nov 03 22:21:08.734797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.734785    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f48daa0-fb13-4d8a-9ec9-5eb17b714143" containerName="revoke-keys"
Nov 03 22:21:08.737920 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.737905    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.792598 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.792565    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:21:08.838951 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.838917    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:21:08.841104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.841082    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"]
Nov 03 22:21:08.858414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.858371    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.858602 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.858423    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvg2h\" (UniqueName: \"kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.858602 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.858575    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.959648 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.959613    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.959827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.959659    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvg2h\" (UniqueName: \"kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.959827 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.959714    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.960365 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.960317    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:08.962239 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:08.962219    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:09.016162 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:09.016084    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvg2h\" (UniqueName: \"kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:09.050512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:09.050473    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:09.427788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:09.427753    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"]
Nov 03 22:21:09.430836 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:21:09.429900    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1e0eec_6552_47cc_9a7c_f4acdcb4b58b.slice/crio-5323d2afb6d99d1a0dce1397a87dcd1e831f00633a713305e22daf14974fc12e WatchSource:0}: Error finding container 5323d2afb6d99d1a0dce1397a87dcd1e831f00633a713305e22daf14974fc12e: Status 404 returned error can't find the container with id 5323d2afb6d99d1a0dce1397a87dcd1e831f00633a713305e22daf14974fc12e
Nov 03 22:21:09.882885 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:09.882853    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p" event={"ID":"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b","Type":"ContainerStarted","Data":"639ff2b19a2c11647f1a8c4c98186db5551342c71d8c6b4db77b62522d53cda1"}
Nov 03 22:21:09.883303 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:09.882913    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p" event={"ID":"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b","Type":"ContainerStarted","Data":"5323d2afb6d99d1a0dce1397a87dcd1e831f00633a713305e22daf14974fc12e"}
Nov 03 22:21:09.996036 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:09.995924    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p" podStartSLOduration=1.995906041 podStartE2EDuration="1.995906041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:09.994186818 +0000 UTC m=+3940.978183392" watchObservedRunningTime="2026-11-03 22:21:09.995906041 +0000 UTC m=+3940.979902601"
Nov 03 22:21:10.489186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.489145    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"]
Nov 03 22:21:10.492811 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.492789    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.550176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.550133    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"]
Nov 03 22:21:10.577569 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.577534    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.577737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.577577    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbd7\" (UniqueName: \"kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.577737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.577724    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.679134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.679033    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.679134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.679081    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbd7\" (UniqueName: \"kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.679405 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.679182    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.679829 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.679810    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.681677 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.681656    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.735688 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.735654    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbd7\" (UniqueName: \"kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7\") pod \"maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.806573 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.806538    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:10.891902 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.891869    5244 generic.go:356] "Generic (PLEG): container finished" podID="7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" containerID="639ff2b19a2c11647f1a8c4c98186db5551342c71d8c6b4db77b62522d53cda1" exitCode=0
Nov 03 22:21:10.892468 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:10.892051    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p" event={"ID":"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b","Type":"ContainerDied","Data":"639ff2b19a2c11647f1a8c4c98186db5551342c71d8c6b4db77b62522d53cda1"}
Nov 03 22:21:11.181379 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:11.181353    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"]
Nov 03 22:21:11.183673 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:21:11.183639    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c2d2d_0a5f_4984_b5d0_27d3ccd14a9d.slice/crio-6bc9316a2953e741c1aa6abb5481bb7076485b317f2b4ab095d71a1f26f85cb9 WatchSource:0}: Error finding container 6bc9316a2953e741c1aa6abb5481bb7076485b317f2b4ab095d71a1f26f85cb9: Status 404 returned error can't find the container with id 6bc9316a2953e741c1aa6abb5481bb7076485b317f2b4ab095d71a1f26f85cb9
Nov 03 22:21:11.899565 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:11.899454    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk" event={"ID":"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d","Type":"ContainerStarted","Data":"804c76c86ed0076f37a05ad2fa6fd588a56a8f4326939c6863624a55560374a8"}
Nov 03 22:21:11.899565 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:11.899528    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk" event={"ID":"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d","Type":"ContainerStarted","Data":"6bc9316a2953e741c1aa6abb5481bb7076485b317f2b4ab095d71a1f26f85cb9"}
Nov 03 22:21:12.014050 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.013994    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk" podStartSLOduration=2.013981308 podStartE2EDuration="2.013981308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:12.012519015 +0000 UTC m=+3942.996515574" watchObservedRunningTime="2026-11-03 22:21:12.013981308 +0000 UTC m=+3942.997977904"
Nov 03 22:21:12.049277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.049251    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:12.099600 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.099561    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca\") pod \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") "
Nov 03 22:21:12.099789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.099662    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h\" (UniqueName: \"kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h\") pod \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") "
Nov 03 22:21:12.099789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.099723    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca\") pod \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\" (UID: \"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b\") "
Nov 03 22:21:12.100205 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.100175    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca" pod "7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" (UID: "7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:12.102273 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.102245    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca" pod "7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" (UID: "7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:12.103157 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.103132    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h" pod "7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" (UID: "7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b"). InnerVolumeSpecName "kube-api-access-jvg2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:12.201131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.201090    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.201131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.201120    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvg2h\" (UniqueName: \"kubernetes.io/projected/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-kube-api-access-jvg2h\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.201131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.201137    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:12.906482 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.906450    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"
Nov 03 22:21:12.906986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.906727    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p" event={"ID":"7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b","Type":"ContainerDied","Data":"5323d2afb6d99d1a0dce1397a87dcd1e831f00633a713305e22daf14974fc12e"}
Nov 03 22:21:12.906986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.906779    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5323d2afb6d99d1a0dce1397a87dcd1e831f00633a713305e22daf14974fc12e"
Nov 03 22:21:12.908510 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.908484    5244 generic.go:356] "Generic (PLEG): container finished" podID="1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" containerID="804c76c86ed0076f37a05ad2fa6fd588a56a8f4326939c6863624a55560374a8" exitCode=0
Nov 03 22:21:12.908655 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:12.908568    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk" event={"ID":"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d","Type":"ContainerDied","Data":"804c76c86ed0076f37a05ad2fa6fd588a56a8f4326939c6863624a55560374a8"}
Nov 03 22:21:13.225789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.225758    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:21:13.316068 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316029    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316120    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316183    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316241    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316275    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316300    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316389    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316430    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316421    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:13.316562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316493    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs\") pod \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\" (UID: \"0ca5f268-88b9-4c99-a0e8-fcf382698cab\") "
Nov 03 22:21:13.316562 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316490    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:13.316855 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316757    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:13.316855 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316779    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:13.317000 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.316963    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:13.317081 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.317062    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.317122 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.317088    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.317122 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.317104    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.317257 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.317191    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.318828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.318803    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:13.318908 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.318888    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "kube-api-access-6lz28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:13.319080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.319059    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:21:13.319145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.319128    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token" pod "0ca5f268-88b9-4c99-a0e8-fcf382698cab" (UID: "0ca5f268-88b9-4c99-a0e8-fcf382698cab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:13.418689 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.418652    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.418689 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.418680    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.418689 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.418691    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.418929 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.418700    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lz28\" (UniqueName: \"kubernetes.io/projected/0ca5f268-88b9-4c99-a0e8-fcf382698cab-kube-api-access-6lz28\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.418929 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.418711    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca5f268-88b9-4c99-a0e8-fcf382698cab-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:13.916246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.916214    5244 generic.go:356] "Generic (PLEG): container finished" podID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerID="2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e" exitCode=0
Nov 03 22:21:13.916734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.916269    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"
Nov 03 22:21:13.916734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.916299    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" event={"ID":"0ca5f268-88b9-4c99-a0e8-fcf382698cab","Type":"ContainerDied","Data":"2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e"}
Nov 03 22:21:13.916734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.916388    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb" event={"ID":"0ca5f268-88b9-4c99-a0e8-fcf382698cab","Type":"ContainerDied","Data":"0988acfe5db107476f320346a529c038e9672c92d528a21443f83422f4c3b098"}
Nov 03 22:21:13.916734 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.916416    5244 scope.go:122] "RemoveContainer" containerID="2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e"
Nov 03 22:21:13.929491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.929470    5244 scope.go:122] "RemoveContainer" containerID="41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"
Nov 03 22:21:13.939752 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.939725    5244 scope.go:122] "RemoveContainer" containerID="2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e"
Nov 03 22:21:13.940130 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:21:13.940102    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e\": container with ID starting with 2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e not found: ID does not exist" containerID="2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e"
Nov 03 22:21:13.940232 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.940140    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e"} err="failed to get container status \"2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e\": rpc error: code = NotFound desc = could not find container \"2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e\": container with ID starting with 2a0ed1eb48258931a1011989f01142113e5f5de5c089577734fd6349a2f4af2e not found: ID does not exist"
Nov 03 22:21:13.940232 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.940161    5244 scope.go:122] "RemoveContainer" containerID="41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"
Nov 03 22:21:13.940534 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:21:13.940505    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb\": container with ID starting with 41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb not found: ID does not exist" containerID="41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"
Nov 03 22:21:13.940589 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:13.940540    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb"} err="failed to get container status \"41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb\": rpc error: code = NotFound desc = could not find container \"41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb\": container with ID starting with 41410d6e2018727d1767258debf9f92c4309a20cefec6da474cde1c33438cbfb not found: ID does not exist"
Nov 03 22:21:14.060121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.060093    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:14.102385 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.102343    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"]
Nov 03 22:21:14.127904 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.127869    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca\") pod \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") "
Nov 03 22:21:14.128146 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.127924    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7\" (UniqueName: \"kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7\") pod \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") "
Nov 03 22:21:14.128146 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.128002    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca\") pod \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\" (UID: \"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d\") "
Nov 03 22:21:14.128778 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.128747    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca" pod "1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" (UID: "1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:14.132524 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.132478    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca" pod "1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" (UID: "1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:14.133825 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.133793    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7" pod "1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" (UID: "1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d"). InnerVolumeSpecName "kube-api-access-dhbd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:14.152257 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.152222    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-29dbd9-openshift-default-7bf4cbc75f-b6bsb"]
Nov 03 22:21:14.229445 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.229412    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.229445 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.229448    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.229929 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.229467    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbd7\" (UniqueName: \"kubernetes.io/projected/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d-kube-api-access-dhbd7\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:14.312806 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.312756    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"]
Nov 03 22:21:14.315127 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.315101    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a49a32-p5g58"]
Nov 03 22:21:14.924295 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.924266    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"
Nov 03 22:21:14.924295 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.924275    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk" event={"ID":"1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d","Type":"ContainerDied","Data":"6bc9316a2953e741c1aa6abb5481bb7076485b317f2b4ab095d71a1f26f85cb9"}
Nov 03 22:21:14.924767 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.924351    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc9316a2953e741c1aa6abb5481bb7076485b317f2b4ab095d71a1f26f85cb9"
Nov 03 22:21:14.970186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.970148    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"]
Nov 03 22:21:14.983601 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:14.983571    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"]
Nov 03 22:21:15.019851 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:15.019818    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"]
Nov 03 22:21:15.139048 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:15.138649    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" path="/var/lib/kubelet/pods/0ca5f268-88b9-4c99-a0e8-fcf382698cab/volumes"
Nov 03 22:21:15.144263 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:15.139315    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cac9d1-1540-4da4-acc6-55b5124b98a7" path="/var/lib/kubelet/pods/51cac9d1-1540-4da4-acc6-55b5124b98a7/volumes"
Nov 03 22:21:15.821032 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:15.821000    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"]
Nov 03 22:21:15.931156 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:15.931122    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"]
Nov 03 22:21:20.822095 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:20.822027    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr" podUID="414b4812-e494-43ed-aecf-0dccdc023f0f" containerName="payload-processing" containerID="cri-o://18292c7297c59e72df15d1cfb4c2de50920f141339b2def439ad7ae667cb4f39" gracePeriod=25
Nov 03 22:21:20.932374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:20.932297    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79" podUID="515dcc36-e0a4-4e79-9e1c-3d5544003ab7" containerName="payload-pre-processing" containerID="cri-o://e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c" gracePeriod=25
Nov 03 22:21:20.971722 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:20.971429    5244 generic.go:356] "Generic (PLEG): container finished" podID="414b4812-e494-43ed-aecf-0dccdc023f0f" containerID="18292c7297c59e72df15d1cfb4c2de50920f141339b2def439ad7ae667cb4f39" exitCode=0
Nov 03 22:21:20.971722 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:20.971501    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr" event={"ID":"414b4812-e494-43ed-aecf-0dccdc023f0f","Type":"ContainerDied","Data":"18292c7297c59e72df15d1cfb4c2de50920f141339b2def439ad7ae667cb4f39"}
Nov 03 22:21:21.150890 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.146795    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:21:21.253228 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.253194    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:21:21.328424 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.328389    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8\" (UniqueName: \"kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8\") pod \"414b4812-e494-43ed-aecf-0dccdc023f0f\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") "
Nov 03 22:21:21.328623 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.328526    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume\") pod \"414b4812-e494-43ed-aecf-0dccdc023f0f\" (UID: \"414b4812-e494-43ed-aecf-0dccdc023f0f\") "
Nov 03 22:21:21.328911 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.328886    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume" pod "414b4812-e494-43ed-aecf-0dccdc023f0f" (UID: "414b4812-e494-43ed-aecf-0dccdc023f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:21.330677 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.330651    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8" pod "414b4812-e494-43ed-aecf-0dccdc023f0f" (UID: "414b4812-e494-43ed-aecf-0dccdc023f0f"). InnerVolumeSpecName "kube-api-access-frnl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:21.429939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.429852    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj\" (UniqueName: \"kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj\") pod \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") "
Nov 03 22:21:21.429939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.429892    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume\") pod \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\" (UID: \"515dcc36-e0a4-4e79-9e1c-3d5544003ab7\") "
Nov 03 22:21:21.430198 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.430182    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414b4812-e494-43ed-aecf-0dccdc023f0f-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:21.430238 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.430203    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnl8\" (UniqueName: \"kubernetes.io/projected/414b4812-e494-43ed-aecf-0dccdc023f0f-kube-api-access-frnl8\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:21.430426 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.430398    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume" pod "515dcc36-e0a4-4e79-9e1c-3d5544003ab7" (UID: "515dcc36-e0a4-4e79-9e1c-3d5544003ab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:21.432106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.432085    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj" pod "515dcc36-e0a4-4e79-9e1c-3d5544003ab7" (UID: "515dcc36-e0a4-4e79-9e1c-3d5544003ab7"). InnerVolumeSpecName "kube-api-access-6nblj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:21.531407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.531370    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nblj\" (UniqueName: \"kubernetes.io/projected/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-kube-api-access-6nblj\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:21.531407 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.531398    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515dcc36-e0a4-4e79-9e1c-3d5544003ab7-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:21.849507 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849468    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"]
Nov 03 22:21:21.849918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849905    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy"
Nov 03 22:21:21.849963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849920    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy"
Nov 03 22:21:21.849963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849933    5244 cpu_manager.go:414] "containerMap: removing container" podUID="414b4812-e494-43ed-aecf-0dccdc023f0f" containerName="payload-processing"
Nov 03 22:21:21.849963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849938    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b4812-e494-43ed-aecf-0dccdc023f0f" containerName="payload-processing"
Nov 03 22:21:21.850056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849962    5244 cpu_manager.go:414] "containerMap: removing container" podUID="515dcc36-e0a4-4e79-9e1c-3d5544003ab7" containerName="payload-pre-processing"
Nov 03 22:21:21.850056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849971    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515dcc36-e0a4-4e79-9e1c-3d5544003ab7" containerName="payload-pre-processing"
Nov 03 22:21:21.850056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849985    5244 cpu_manager.go:414] "containerMap: removing container" podUID="7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" containerName="revoke-keys"
Nov 03 22:21:21.850056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849990    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" containerName="revoke-keys"
Nov 03 22:21:21.850056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.849997    5244 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" containerName="revoke-keys"
Nov 03 22:21:21.850056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850001    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" containerName="revoke-keys"
Nov 03 22:21:21.850245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850064    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" containerName="revoke-keys"
Nov 03 22:21:21.850245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850077    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy"
Nov 03 22:21:21.850245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850082    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="515dcc36-e0a4-4e79-9e1c-3d5544003ab7" containerName="payload-pre-processing"
Nov 03 22:21:21.850245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850089    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy"
Nov 03 22:21:21.850245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850095    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="414b4812-e494-43ed-aecf-0dccdc023f0f" containerName="payload-processing"
Nov 03 22:21:21.850245 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.850101    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" containerName="revoke-keys"
Nov 03 22:21:21.853479 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.853460    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:21.911121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.911091    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:21:21.956642 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.956611    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:21:21.958956 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.958928    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"]
Nov 03 22:21:21.978460 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.978430    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"
Nov 03 22:21:21.978678 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.978435    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr" event={"ID":"414b4812-e494-43ed-aecf-0dccdc023f0f","Type":"ContainerDied","Data":"a01fb5c2eefb6c85f27278d3a2a81e173309b814f2785cf94de9ff2b30fa83a7"}
Nov 03 22:21:21.978678 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.978572    5244 scope.go:122] "RemoveContainer" containerID="18292c7297c59e72df15d1cfb4c2de50920f141339b2def439ad7ae667cb4f39"
Nov 03 22:21:21.980080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.979833    5244 generic.go:356] "Generic (PLEG): container finished" podID="515dcc36-e0a4-4e79-9e1c-3d5544003ab7" containerID="e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c" exitCode=0
Nov 03 22:21:21.980080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.979880    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"
Nov 03 22:21:21.980080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.979931    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79" event={"ID":"515dcc36-e0a4-4e79-9e1c-3d5544003ab7","Type":"ContainerDied","Data":"e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c"}
Nov 03 22:21:21.980080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.979979    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79" event={"ID":"515dcc36-e0a4-4e79-9e1c-3d5544003ab7","Type":"ContainerDied","Data":"dc499d0eb7ae8f30550455955943ba45a984fc40de20949e9b5e05f57721447d"}
Nov 03 22:21:21.989924 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.989897    5244 scope.go:122] "RemoveContainer" containerID="e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c"
Nov 03 22:21:21.999161 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.999142    5244 scope.go:122] "RemoveContainer" containerID="e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c"
Nov 03 22:21:21.999479 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:21:21.999457    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c\": container with ID starting with e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c not found: ID does not exist" containerID="e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c"
Nov 03 22:21:21.999547 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:21.999491    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c"} err="failed to get container status \"e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c\": rpc error: code = NotFound desc = could not find container \"e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c\": container with ID starting with e08a3acfe347461d3e0d7ca7929980c128fa12be524d435aec6bef896480388c not found: ID does not exist"
Nov 03 22:21:22.036227 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.036193    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frcm\" (UniqueName: \"kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.036401 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.036237    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.036401 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.036361    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.137305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.137223    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frcm\" (UniqueName: \"kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.137305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.137274    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.137543 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.137322    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.138026 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.137984    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.140179 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.140151    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.140856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.140823    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"]
Nov 03 22:21:22.145914 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.145891    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-48ae0a-69d74b48b6-xx6sr"]
Nov 03 22:21:22.194104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.194071    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frcm\" (UniqueName: \"kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.254490 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.254455    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"]
Nov 03 22:21:22.257634 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.257607    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-48ae0a-66dcc5cb4c-pcp79"]
Nov 03 22:21:22.435894 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:21:22.435861    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:21:22.465717 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.465688    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:22.839241 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.839214    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"]
Nov 03 22:21:22.841443 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:21:22.841406    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae489d5_cd48_4c19_b0e6_0428ece95552.slice/crio-52daed5155f3c6a5fb3f13d44781fb032344a7bc32d0cdb3403a2e2d422c4157 WatchSource:0}: Error finding container 52daed5155f3c6a5fb3f13d44781fb032344a7bc32d0cdb3403a2e2d422c4157: Status 404 returned error can't find the container with id 52daed5155f3c6a5fb3f13d44781fb032344a7bc32d0cdb3403a2e2d422c4157
Nov 03 22:21:22.989388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:22.989354    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp" event={"ID":"6ae489d5-cd48-4c19-b0e6-0428ece95552","Type":"ContainerStarted","Data":"52daed5155f3c6a5fb3f13d44781fb032344a7bc32d0cdb3403a2e2d422c4157"}
Nov 03 22:21:23.135969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:23.135939    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414b4812-e494-43ed-aecf-0dccdc023f0f" path="/var/lib/kubelet/pods/414b4812-e494-43ed-aecf-0dccdc023f0f/volumes"
Nov 03 22:21:23.136398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:23.136383    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515dcc36-e0a4-4e79-9e1c-3d5544003ab7" path="/var/lib/kubelet/pods/515dcc36-e0a4-4e79-9e1c-3d5544003ab7/volumes"
Nov 03 22:21:23.998268 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:23.998177    5244 generic.go:356] "Generic (PLEG): container finished" podID="6ae489d5-cd48-4c19-b0e6-0428ece95552" containerID="60127da5f63277c207d5e75d7928fa9e9a3f39f81a3af9628bcebee7b9fa850f" exitCode=0
Nov 03 22:21:23.998695 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:23.998270    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp" event={"ID":"6ae489d5-cd48-4c19-b0e6-0428ece95552","Type":"ContainerDied","Data":"60127da5f63277c207d5e75d7928fa9e9a3f39f81a3af9628bcebee7b9fa850f"}
Nov 03 22:21:25.170187 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.170161    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:25.275597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.275501    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm\" (UniqueName: \"kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm\") pod \"6ae489d5-cd48-4c19-b0e6-0428ece95552\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") "
Nov 03 22:21:25.275597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.275568    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca\") pod \"6ae489d5-cd48-4c19-b0e6-0428ece95552\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") "
Nov 03 22:21:25.275810 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.275605    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca\") pod \"6ae489d5-cd48-4c19-b0e6-0428ece95552\" (UID: \"6ae489d5-cd48-4c19-b0e6-0428ece95552\") "
Nov 03 22:21:25.276050 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.276026    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca" pod "6ae489d5-cd48-4c19-b0e6-0428ece95552" (UID: "6ae489d5-cd48-4c19-b0e6-0428ece95552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:25.277839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.277802    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm" pod "6ae489d5-cd48-4c19-b0e6-0428ece95552" (UID: "6ae489d5-cd48-4c19-b0e6-0428ece95552"). InnerVolumeSpecName "kube-api-access-8frcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:25.278111 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.278002    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca" pod "6ae489d5-cd48-4c19-b0e6-0428ece95552" (UID: "6ae489d5-cd48-4c19-b0e6-0428ece95552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:25.376978 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.376938    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frcm\" (UniqueName: \"kubernetes.io/projected/6ae489d5-cd48-4c19-b0e6-0428ece95552-kube-api-access-8frcm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:25.376978 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.376980    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:25.377229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:25.376995    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae489d5-cd48-4c19-b0e6-0428ece95552-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:26.012790 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:26.012763    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"
Nov 03 22:21:26.012979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:26.012801    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp" event={"ID":"6ae489d5-cd48-4c19-b0e6-0428ece95552","Type":"ContainerDied","Data":"52daed5155f3c6a5fb3f13d44781fb032344a7bc32d0cdb3403a2e2d422c4157"}
Nov 03 22:21:26.012979 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:26.012846    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52daed5155f3c6a5fb3f13d44781fb032344a7bc32d0cdb3403a2e2d422c4157"
Nov 03 22:21:29.991656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:29.991597    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main" containerID="cri-o://bf43c22949e6eac05a57f0ea800ff1a3257243a813abcd4c1aa4f1d2d2148393" gracePeriod=45
Nov 03 22:21:30.011791 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.011761    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main" containerID="cri-o://709c5c952ac1b55e6f5d49a6df16ed008fb43fa6ed65b66a36ee6ea594fe6f5c" gracePeriod=45
Nov 03 22:21:30.042728 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.042688    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main" containerID="cri-o://41e31d4e5604901fb0cd9f04a132e539db6c60c92fb9b996b4cd5377ade0fe43" gracePeriod=45
Nov 03 22:21:30.297228 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.297200    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 22:21:30.326297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326271    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs\") pod \"67a43564-488d-43e3-a765-9fda2fb7f0eb\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") "
Nov 03 22:21:30.326493 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326363    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home\") pod \"67a43564-488d-43e3-a765-9fda2fb7f0eb\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") "
Nov 03 22:21:30.326493 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326413    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir\") pod \"67a43564-488d-43e3-a765-9fda2fb7f0eb\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") "
Nov 03 22:21:30.326493 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326459    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm\") pod \"67a43564-488d-43e3-a765-9fda2fb7f0eb\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") "
Nov 03 22:21:30.326667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326499    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9\" (UniqueName: \"kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9\") pod \"67a43564-488d-43e3-a765-9fda2fb7f0eb\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") "
Nov 03 22:21:30.326667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326550    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache\") pod \"67a43564-488d-43e3-a765-9fda2fb7f0eb\" (UID: \"67a43564-488d-43e3-a765-9fda2fb7f0eb\") "
Nov 03 22:21:30.326793 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326771    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 22:21:30.326887 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.326860    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir" pod "67a43564-488d-43e3-a765-9fda2fb7f0eb" (UID: "67a43564-488d-43e3-a765-9fda2fb7f0eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.327103 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.327075    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache" pod "67a43564-488d-43e3-a765-9fda2fb7f0eb" (UID: "67a43564-488d-43e3-a765-9fda2fb7f0eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.327246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.327227    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home" pod "67a43564-488d-43e3-a765-9fda2fb7f0eb" (UID: "67a43564-488d-43e3-a765-9fda2fb7f0eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.328815 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.328791    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs" pod "67a43564-488d-43e3-a765-9fda2fb7f0eb" (UID: "67a43564-488d-43e3-a765-9fda2fb7f0eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:30.328964 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.328943    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm" pod "67a43564-488d-43e3-a765-9fda2fb7f0eb" (UID: "67a43564-488d-43e3-a765-9fda2fb7f0eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.329071 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.329057    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9" pod "67a43564-488d-43e3-a765-9fda2fb7f0eb" (UID: "67a43564-488d-43e3-a765-9fda2fb7f0eb"). InnerVolumeSpecName "kube-api-access-6wwf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:30.331995 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.331974    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 22:21:30.427675 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427632    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs\" (UniqueName: \"kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs\") pod \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") "
Nov 03 22:21:30.427859 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427715    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir\") pod \"0358e895-256c-43ec-9c68-38bb70e633e4\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") "
Nov 03 22:21:30.427859 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427793    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir\") pod \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") "
Nov 03 22:21:30.427945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427863    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs\") pod \"0358e895-256c-43ec-9c68-38bb70e633e4\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") "
Nov 03 22:21:30.427945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427895    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2\" (UniqueName: \"kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2\") pod \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") "
Nov 03 22:21:30.427945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427918    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home\") pod \"0358e895-256c-43ec-9c68-38bb70e633e4\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") "
Nov 03 22:21:30.428080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.427987    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache\") pod \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") "
Nov 03 22:21:30.428080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428014    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm\") pod \"0358e895-256c-43ec-9c68-38bb70e633e4\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") "
Nov 03 22:21:30.428080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428021    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir" pod "0358e895-256c-43ec-9c68-38bb70e633e4" (UID: "0358e895-256c-43ec-9c68-38bb70e633e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.428080 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428053    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache\") pod \"0358e895-256c-43ec-9c68-38bb70e633e4\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") "
Nov 03 22:21:30.428317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428081    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg\" (UniqueName: \"kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg\") pod \"0358e895-256c-43ec-9c68-38bb70e633e4\" (UID: \"0358e895-256c-43ec-9c68-38bb70e633e4\") "
Nov 03 22:21:30.428317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428113    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home\") pod \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") "
Nov 03 22:21:30.428317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428162    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm\") pod \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\" (UID: \"d568b2bb-57fb-46c1-b824-7e249a7cfb49\") "
Nov 03 22:21:30.428317 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428231    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache" pod "d568b2bb-57fb-46c1-b824-7e249a7cfb49" (UID: "d568b2bb-57fb-46c1-b824-7e249a7cfb49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.428608 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428583    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwf9\" (UniqueName: \"kubernetes.io/projected/67a43564-488d-43e3-a765-9fda2fb7f0eb-kube-api-access-6wwf9\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.428703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428616    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.428703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428634    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.428703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428690    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a43564-488d-43e3-a765-9fda2fb7f0eb-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428704    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428718    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428731    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428745    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a43564-488d-43e3-a765-9fda2fb7f0eb-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428745    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache" pod "0358e895-256c-43ec-9c68-38bb70e633e4" (UID: "0358e895-256c-43ec-9c68-38bb70e633e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.429217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.428968    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir" pod "d568b2bb-57fb-46c1-b824-7e249a7cfb49" (UID: "d568b2bb-57fb-46c1-b824-7e249a7cfb49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.429674 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.429647    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home" pod "d568b2bb-57fb-46c1-b824-7e249a7cfb49" (UID: "d568b2bb-57fb-46c1-b824-7e249a7cfb49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.430591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.430553    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs" pod "d568b2bb-57fb-46c1-b824-7e249a7cfb49" (UID: "d568b2bb-57fb-46c1-b824-7e249a7cfb49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:30.430857 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.430835    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home" pod "0358e895-256c-43ec-9c68-38bb70e633e4" (UID: "0358e895-256c-43ec-9c68-38bb70e633e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.431399 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.431375    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm" pod "0358e895-256c-43ec-9c68-38bb70e633e4" (UID: "0358e895-256c-43ec-9c68-38bb70e633e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.431506 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.431452    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2" pod "d568b2bb-57fb-46c1-b824-7e249a7cfb49" (UID: "d568b2bb-57fb-46c1-b824-7e249a7cfb49"). InnerVolumeSpecName "kube-api-access-ntfc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:30.432986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.432957    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg" pod "0358e895-256c-43ec-9c68-38bb70e633e4" (UID: "0358e895-256c-43ec-9c68-38bb70e633e4"). InnerVolumeSpecName "kube-api-access-tm9vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:30.433091 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.433004    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm" pod "d568b2bb-57fb-46c1-b824-7e249a7cfb49" (UID: "d568b2bb-57fb-46c1-b824-7e249a7cfb49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:21:30.433189 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.433166    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs" pod "0358e895-256c-43ec-9c68-38bb70e633e4" (UID: "0358e895-256c-43ec-9c68-38bb70e633e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:30.529783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529687    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.529783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529724    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.529783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529742    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm9vg\" (UniqueName: \"kubernetes.io/projected/0358e895-256c-43ec-9c68-38bb70e633e4-kube-api-access-tm9vg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.529783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529759    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.529783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529773    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.529783 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529786    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.530134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529800    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d568b2bb-57fb-46c1-b824-7e249a7cfb49-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.530134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529812    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358e895-256c-43ec-9c68-38bb70e633e4-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.530134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529827    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntfc2\" (UniqueName: \"kubernetes.io/projected/d568b2bb-57fb-46c1-b824-7e249a7cfb49-kube-api-access-ntfc2\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.530134 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.529841    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358e895-256c-43ec-9c68-38bb70e633e4-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:30.843106 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.843007    5244 scope.go:122] "RemoveContainer" containerID="9ae65c3622ef3883d4f246b0c9419cd1ac53413e6d479b1818356562af3e5005"
Nov 03 22:21:30.853037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.853015    5244 scope.go:122] "RemoveContainer" containerID="709c5c952ac1b55e6f5d49a6df16ed008fb43fa6ed65b66a36ee6ea594fe6f5c"
Nov 03 22:21:30.862120 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.862093    5244 scope.go:122] "RemoveContainer" containerID="41e31d4e5604901fb0cd9f04a132e539db6c60c92fb9b996b4cd5377ade0fe43"
Nov 03 22:21:30.870856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:30.870832    5244 scope.go:122] "RemoveContainer" containerID="bf43c22949e6eac05a57f0ea800ff1a3257243a813abcd4c1aa4f1d2d2148393"
Nov 03 22:21:31.042234 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042197    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" event={"ID":"d568b2bb-57fb-46c1-b824-7e249a7cfb49","Type":"ContainerDied","Data":"41e31d4e5604901fb0cd9f04a132e539db6c60c92fb9b996b4cd5377ade0fe43"}
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042251    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f" event={"ID":"d568b2bb-57fb-46c1-b824-7e249a7cfb49","Type":"ContainerDied","Data":"aabf59e5647bb0163e9388e2aafc26c039209ed298482ed9a09561267d724d54"}
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042267    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042273    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042281    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" event={"ID":"0358e895-256c-43ec-9c68-38bb70e633e4","Type":"ContainerDied","Data":"709c5c952ac1b55e6f5d49a6df16ed008fb43fa6ed65b66a36ee6ea594fe6f5c"}
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042345    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6" event={"ID":"0358e895-256c-43ec-9c68-38bb70e633e4","Type":"ContainerDied","Data":"919fdb0c4613c2a7d866917d638e4dfbd5ee89adbf1e99f32613c46feed7e57a"}
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042369    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" event={"ID":"67a43564-488d-43e3-a765-9fda2fb7f0eb","Type":"ContainerDied","Data":"bf43c22949e6eac05a57f0ea800ff1a3257243a813abcd4c1aa4f1d2d2148393"}
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042385    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4" event={"ID":"67a43564-488d-43e3-a765-9fda2fb7f0eb","Type":"ContainerDied","Data":"6e25456117e306b47df93dcda3bc55259bf0196dec9691b87a61babfffec2bdd"}
Nov 03 22:21:31.042667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.042285    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"
Nov 03 22:21:31.332362 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.332314    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"]
Nov 03 22:21:31.337315 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.337288    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/distinct-2-w1-f463c9-kserve-54446bcd54-hqhq4"]
Nov 03 22:21:31.446434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.445919    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"]
Nov 03 22:21:31.447743 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.447718    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/simulator-w1-f463c9-kserve-849f45855-47f4f"]
Nov 03 22:21:31.615131 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.615040    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"]
Nov 03 22:21:31.619856 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:31.619831    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f463c9/premium-w1-f463c9-kserve-7dd55c6554-v2tr6"]
Nov 03 22:21:33.135019 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:33.134980    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" path="/var/lib/kubelet/pods/0358e895-256c-43ec-9c68-38bb70e633e4/volumes"
Nov 03 22:21:33.135428 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:33.135410    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" path="/var/lib/kubelet/pods/67a43564-488d-43e3-a765-9fda2fb7f0eb/volumes"
Nov 03 22:21:33.135763 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:33.135749    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" path="/var/lib/kubelet/pods/d568b2bb-57fb-46c1-b824-7e249a7cfb49/volumes"
Nov 03 22:21:44.416417 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.414777    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"]
Nov 03 22:21:44.422471 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422428    5244 cpu_manager.go:414] "containerMap: removing container" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main"
Nov 03 22:21:44.422471 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422467    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422484    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422491    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca5f268-88b9-4c99-a0e8-fcf382698cab" containerName="istio-proxy"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422518    5244 cpu_manager.go:414] "containerMap: removing container" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422523    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422533    5244 cpu_manager.go:414] "containerMap: removing container" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422541    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422550    5244 cpu_manager.go:414] "containerMap: removing container" podUID="6ae489d5-cd48-4c19-b0e6-0428ece95552" containerName="revoke-keys"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422555    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae489d5-cd48-4c19-b0e6-0428ece95552" containerName="revoke-keys"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422678    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="0358e895-256c-43ec-9c68-38bb70e633e4" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422686    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a43564-488d-43e3-a765-9fda2fb7f0eb" containerName="main"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422695    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae489d5-cd48-4c19-b0e6-0428ece95552" containerName="revoke-keys"
Nov 03 22:21:44.422720 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.422705    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="d568b2bb-57fb-46c1-b824-7e249a7cfb49" containerName="main"
Nov 03 22:21:44.427614 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.427587    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.467910 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.467874    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"]
Nov 03 22:21:44.472798 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.472765    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjmr6\" (UniqueName: \"kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.472970 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.472895    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.473062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.473015    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.514960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.514926    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:21:44.515143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.515019    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:21:44.573903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.573867    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.574078 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.573939    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjmr6\" (UniqueName: \"kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.574078 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.574014    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.574811 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.574781    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.576591 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.576571    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.630518 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.630484    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjmr6\" (UniqueName: \"kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6\") pod \"maas-api-revoke-sub-3f0cbed5-xnqv5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:44.749128 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:44.749085    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:45.127525 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:45.127426    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"]
Nov 03 22:21:45.130644 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:21:45.130614    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c83d91_edc1_4c87_bd80_9918a9ebe7f5.slice/crio-35b917f707ad0e2b7d973ee98a5fc98239328303f6455276a38501fce948e4fb WatchSource:0}: Error finding container 35b917f707ad0e2b7d973ee98a5fc98239328303f6455276a38501fce948e4fb: Status 404 returned error can't find the container with id 35b917f707ad0e2b7d973ee98a5fc98239328303f6455276a38501fce948e4fb
Nov 03 22:21:45.292605 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:45.292557    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"]
Nov 03 22:21:45.294343 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:45.294302    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-87e1cf-4w86l"]
Nov 03 22:21:46.126298 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:46.126261    5244 generic.go:356] "Generic (PLEG): container finished" podID="83c83d91-edc1-4c87-bd80-9918a9ebe7f5" containerID="4ae883e9f30d47c91d462855ef416a521cfb5453955d806fbdcacf07c8932cbe" exitCode=0
Nov 03 22:21:46.126802 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:46.126376    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5" event={"ID":"83c83d91-edc1-4c87-bd80-9918a9ebe7f5","Type":"ContainerDied","Data":"4ae883e9f30d47c91d462855ef416a521cfb5453955d806fbdcacf07c8932cbe"}
Nov 03 22:21:46.126802 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:46.126452    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5" event={"ID":"83c83d91-edc1-4c87-bd80-9918a9ebe7f5","Type":"ContainerStarted","Data":"35b917f707ad0e2b7d973ee98a5fc98239328303f6455276a38501fce948e4fb"}
Nov 03 22:21:47.135730 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.135701    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9d6338-5031-4546-854a-de49316b03b8" path="/var/lib/kubelet/pods/8a9d6338-5031-4546-854a-de49316b03b8/volumes"
Nov 03 22:21:47.278398 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.278373    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:47.400501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.400411    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca\") pod \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") "
Nov 03 22:21:47.400501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.400470    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca\") pod \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") "
Nov 03 22:21:47.400749 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.400588    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6\" (UniqueName: \"kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6\") pod \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\" (UID: \"83c83d91-edc1-4c87-bd80-9918a9ebe7f5\") "
Nov 03 22:21:47.400828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.400803    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca" pod "83c83d91-edc1-4c87-bd80-9918a9ebe7f5" (UID: "83c83d91-edc1-4c87-bd80-9918a9ebe7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:47.400964 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.400949    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:47.403468 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.403443    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca" pod "83c83d91-edc1-4c87-bd80-9918a9ebe7f5" (UID: "83c83d91-edc1-4c87-bd80-9918a9ebe7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:47.403595 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.403523    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6" pod "83c83d91-edc1-4c87-bd80-9918a9ebe7f5" (UID: "83c83d91-edc1-4c87-bd80-9918a9ebe7f5"). InnerVolumeSpecName "kube-api-access-rjmr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:47.501780 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.501746    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjmr6\" (UniqueName: \"kubernetes.io/projected/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-kube-api-access-rjmr6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:47.501780 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:47.501776    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c83d91-edc1-4c87-bd80-9918a9ebe7f5-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:48.139817 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:48.139784    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5"
Nov 03 22:21:48.140248 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:48.139784    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f0cbed5-xnqv5" event={"ID":"83c83d91-edc1-4c87-bd80-9918a9ebe7f5","Type":"ContainerDied","Data":"35b917f707ad0e2b7d973ee98a5fc98239328303f6455276a38501fce948e4fb"}
Nov 03 22:21:48.140248 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:48.139909    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b917f707ad0e2b7d973ee98a5fc98239328303f6455276a38501fce948e4fb"
Nov 03 22:21:52.435985 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:21:52.435947    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:21:53.406677 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.406641    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"]
Nov 03 22:21:53.407158 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.407137    5244 cpu_manager.go:414] "containerMap: removing container" podUID="83c83d91-edc1-4c87-bd80-9918a9ebe7f5" containerName="revoke-keys"
Nov 03 22:21:53.407246 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.407159    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c83d91-edc1-4c87-bd80-9918a9ebe7f5" containerName="revoke-keys"
Nov 03 22:21:53.407307 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.407281    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c83d91-edc1-4c87-bd80-9918a9ebe7f5" containerName="revoke-keys"
Nov 03 22:21:53.410596 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.410574    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.468151 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.468111    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"]
Nov 03 22:21:53.514774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.514740    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:21:53.514993 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.514882    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:21:53.569136 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.569095    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rc2\" (UniqueName: \"kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.569376 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.569143    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.569376 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.569276    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.670201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.670109    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.670201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.670196    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rc2\" (UniqueName: \"kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.670491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.670221    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.670862 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.670833    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.673009 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.672983    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:53.727125 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:53.727087    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rc2\" (UniqueName: \"kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:54.022649 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:54.022620    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:54.403352 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:54.403312    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"]
Nov 03 22:21:54.405237 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:21:54.405209    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cf5517_0463_4d69_9384_f69c91e7505f.slice/crio-9afb491bb08bde6d91fc463c96e36e3b8715ebba807ce0edd257bd58e1d03ede WatchSource:0}: Error finding container 9afb491bb08bde6d91fc463c96e36e3b8715ebba807ce0edd257bd58e1d03ede: Status 404 returned error can't find the container with id 9afb491bb08bde6d91fc463c96e36e3b8715ebba807ce0edd257bd58e1d03ede
Nov 03 22:21:55.180667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:55.180574    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k" event={"ID":"86cf5517-0463-4d69-9384-f69c91e7505f","Type":"ContainerStarted","Data":"3982b747b9dce1c6b57fe85befbb903bfd2a8abfc7a0382e4d34289adcc8f5b3"}
Nov 03 22:21:55.180667 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:55.180628    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k" event={"ID":"86cf5517-0463-4d69-9384-f69c91e7505f","Type":"ContainerStarted","Data":"9afb491bb08bde6d91fc463c96e36e3b8715ebba807ce0edd257bd58e1d03ede"}
Nov 03 22:21:55.294382 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:55.294308    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k" podStartSLOduration=2.294293051 podStartE2EDuration="2.294293051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:55.292852999 +0000 UTC m=+3986.276849560" watchObservedRunningTime="2026-11-03 22:21:55.294293051 +0000 UTC m=+3986.278289610"
Nov 03 22:21:56.187797 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:56.187766    5244 generic.go:356] "Generic (PLEG): container finished" podID="86cf5517-0463-4d69-9384-f69c91e7505f" containerID="3982b747b9dce1c6b57fe85befbb903bfd2a8abfc7a0382e4d34289adcc8f5b3" exitCode=0
Nov 03 22:21:56.188226 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:56.187864    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k" event={"ID":"86cf5517-0463-4d69-9384-f69c91e7505f","Type":"ContainerDied","Data":"3982b747b9dce1c6b57fe85befbb903bfd2a8abfc7a0382e4d34289adcc8f5b3"}
Nov 03 22:21:56.856206 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:56.856163    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"]
Nov 03 22:21:56.859178 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:56.859149    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-48ae0a-j5dtp"]
Nov 03 22:21:57.136085 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.135995    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f48daa0-fb13-4d8a-9ec9-5eb17b714143" path="/var/lib/kubelet/pods/8f48daa0-fb13-4d8a-9ec9-5eb17b714143/volumes"
Nov 03 22:21:57.338687 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.338659    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:57.411663 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.411579    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca\") pod \"86cf5517-0463-4d69-9384-f69c91e7505f\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") "
Nov 03 22:21:57.411663 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.411618    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca\") pod \"86cf5517-0463-4d69-9384-f69c91e7505f\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") "
Nov 03 22:21:57.411911 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.411727    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2\" (UniqueName: \"kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2\") pod \"86cf5517-0463-4d69-9384-f69c91e7505f\" (UID: \"86cf5517-0463-4d69-9384-f69c91e7505f\") "
Nov 03 22:21:57.412056 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.412034    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca" pod "86cf5517-0463-4d69-9384-f69c91e7505f" (UID: "86cf5517-0463-4d69-9384-f69c91e7505f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:21:57.413879 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.413854    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca" pod "86cf5517-0463-4d69-9384-f69c91e7505f" (UID: "86cf5517-0463-4d69-9384-f69c91e7505f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:21:57.413996 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.413899    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2" pod "86cf5517-0463-4d69-9384-f69c91e7505f" (UID: "86cf5517-0463-4d69-9384-f69c91e7505f"). InnerVolumeSpecName "kube-api-access-g6rc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:21:57.513024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.512986    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:57.513024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.513020    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cf5517-0463-4d69-9384-f69c91e7505f-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:57.513024 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:57.513030    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rc2\" (UniqueName: \"kubernetes.io/projected/86cf5517-0463-4d69-9384-f69c91e7505f-kube-api-access-g6rc2\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:21:58.199758 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:58.199728    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"
Nov 03 22:21:58.199955 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:58.199722    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k" event={"ID":"86cf5517-0463-4d69-9384-f69c91e7505f","Type":"ContainerDied","Data":"9afb491bb08bde6d91fc463c96e36e3b8715ebba807ce0edd257bd58e1d03ede"}
Nov 03 22:21:58.199955 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:21:58.199828    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afb491bb08bde6d91fc463c96e36e3b8715ebba807ce0edd257bd58e1d03ede"
Nov 03 22:22:02.176446 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:02.176410    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:22:03.223166 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:03.223113    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"]
Nov 03 22:22:03.272512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:03.272479    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-0d45ba-qkv7p"]
Nov 03 22:22:05.137097 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:05.137065    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b" path="/var/lib/kubelet/pods/7c1e0eec-6552-47cc-9a7c-f4acdcb4b58b/volumes"
Nov 03 22:22:08.043806 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:08.043773    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"]
Nov 03 22:22:08.093083 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:08.093041    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1da14cb-j45x4"]
Nov 03 22:22:09.136057 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:09.136021    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5837fcb5-7b15-4e6a-bda0-0e423fa72963" path="/var/lib/kubelet/pods/5837fcb5-7b15-4e6a-bda0-0e423fa72963/volumes"
Nov 03 22:22:14.923248 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:14.923207    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"]
Nov 03 22:22:14.972504 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:14.972468    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75f577-z29pp"]
Nov 03 22:22:15.135567 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:15.135534    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae489d5-cd48-4c19-b0e6-0428ece95552" path="/var/lib/kubelet/pods/6ae489d5-cd48-4c19-b0e6-0428ece95552/volumes"
Nov 03 22:22:17.651926 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:17.651893    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"]
Nov 03 22:22:17.762107 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:17.762070    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"]
Nov 03 22:22:19.043479 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:19.043442    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"]
Nov 03 22:22:19.045818 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:19.045792    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b4f0a4-4mq2s"]
Nov 03 22:22:19.136961 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:19.136924    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1dbf58-c446-42c2-9690-c7208af622b0" path="/var/lib/kubelet/pods/2f1dbf58-c446-42c2-9690-c7208af622b0/volumes"
Nov 03 22:22:22.438112 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:22.438072    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:22:22.652969 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.652927    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf" podUID="c81d8fbf-b920-45cd-94b7-d7f97aebaa97" containerName="payload-processing" containerID="cri-o://1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca" gracePeriod=25
Nov 03 22:22:22.763509 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.763378    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c" podUID="221a1e8e-d593-4e40-b27a-b201d76c7e46" containerName="payload-pre-processing" containerID="cri-o://d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d" gracePeriod=25
Nov 03 22:22:22.900680 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.900645    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:22:22.987757 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.987717    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4\" (UniqueName: \"kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4\") pod \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") "
Nov 03 22:22:22.987972 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.987791    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume\") pod \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\" (UID: \"c81d8fbf-b920-45cd-94b7-d7f97aebaa97\") "
Nov 03 22:22:22.988190 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.988161    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume" pod "c81d8fbf-b920-45cd-94b7-d7f97aebaa97" (UID: "c81d8fbf-b920-45cd-94b7-d7f97aebaa97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:22.990576 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:22.990549    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4" pod "c81d8fbf-b920-45cd-94b7-d7f97aebaa97" (UID: "c81d8fbf-b920-45cd-94b7-d7f97aebaa97"). InnerVolumeSpecName "kube-api-access-9ckx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:23.010930 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.010904    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:22:23.089505 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.089420    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf\" (UniqueName: \"kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf\") pod \"221a1e8e-d593-4e40-b27a-b201d76c7e46\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") "
Nov 03 22:22:23.089505 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.089469    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume\") pod \"221a1e8e-d593-4e40-b27a-b201d76c7e46\" (UID: \"221a1e8e-d593-4e40-b27a-b201d76c7e46\") "
Nov 03 22:22:23.089725 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.089673    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckx4\" (UniqueName: \"kubernetes.io/projected/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-kube-api-access-9ckx4\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:23.089725 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.089684    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c81d8fbf-b920-45cd-94b7-d7f97aebaa97-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:23.089876 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.089850    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume" pod "221a1e8e-d593-4e40-b27a-b201d76c7e46" (UID: "221a1e8e-d593-4e40-b27a-b201d76c7e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:23.091703 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.091686    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf" pod "221a1e8e-d593-4e40-b27a-b201d76c7e46" (UID: "221a1e8e-d593-4e40-b27a-b201d76c7e46"). InnerVolumeSpecName "kube-api-access-wzcgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:23.191285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.191249    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcgf\" (UniqueName: \"kubernetes.io/projected/221a1e8e-d593-4e40-b27a-b201d76c7e46-kube-api-access-wzcgf\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:23.191285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.191286    5244 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a1e8e-d593-4e40-b27a-b201d76c7e46-plugins-config-volume\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:23.346406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.346295    5244 generic.go:356] "Generic (PLEG): container finished" podID="221a1e8e-d593-4e40-b27a-b201d76c7e46" containerID="d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d" exitCode=0
Nov 03 22:22:23.346406 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.346377    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c" event={"ID":"221a1e8e-d593-4e40-b27a-b201d76c7e46","Type":"ContainerDied","Data":"d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d"}
Nov 03 22:22:23.346660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.346444    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c" event={"ID":"221a1e8e-d593-4e40-b27a-b201d76c7e46","Type":"ContainerDied","Data":"f2e2affb997709162c608aab662c464868d532f6203b827f7a9409896705d721"}
Nov 03 22:22:23.346660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.346374    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"
Nov 03 22:22:23.346660 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.346475    5244 scope.go:122] "RemoveContainer" containerID="d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d"
Nov 03 22:22:23.348737 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.348681    5244 generic.go:356] "Generic (PLEG): container finished" podID="c81d8fbf-b920-45cd-94b7-d7f97aebaa97" containerID="1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca" exitCode=0
Nov 03 22:22:23.349060 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.349032    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf" event={"ID":"c81d8fbf-b920-45cd-94b7-d7f97aebaa97","Type":"ContainerDied","Data":"1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca"}
Nov 03 22:22:23.349159 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.349070    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"
Nov 03 22:22:23.349159 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.349080    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf" event={"ID":"c81d8fbf-b920-45cd-94b7-d7f97aebaa97","Type":"ContainerDied","Data":"32dea064bba0a4bf4816b2ed7bac36d96198f03b3fa19f868bee033258479e2a"}
Nov 03 22:22:23.357360 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.357313    5244 scope.go:122] "RemoveContainer" containerID="d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d"
Nov 03 22:22:23.357730 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:23.357705    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d\": container with ID starting with d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d not found: ID does not exist" containerID="d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d"
Nov 03 22:22:23.357786 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.357745    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d"} err="failed to get container status \"d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d\": rpc error: code = NotFound desc = could not find container \"d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d\": container with ID starting with d4acbfc72e05cfc4c766d6511161c86a6b279e198ccee28395f4a330eea6106d not found: ID does not exist"
Nov 03 22:22:23.357786 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.357772    5244 scope.go:122] "RemoveContainer" containerID="1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca"
Nov 03 22:22:23.367277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.367258    5244 scope.go:122] "RemoveContainer" containerID="1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca"
Nov 03 22:22:23.367703 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:23.367680    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca\": container with ID starting with 1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca not found: ID does not exist" containerID="1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca"
Nov 03 22:22:23.367759 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.367712    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca"} err="failed to get container status \"1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca\": rpc error: code = NotFound desc = could not find container \"1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca\": container with ID starting with 1a6fd4a04037102190c06ee04e1ee07f43c5a88252398071d24b2d92b73ad9ca not found: ID does not exist"
Nov 03 22:22:23.509507 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.509472    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"]
Nov 03 22:22:23.512032 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.512001    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e15bbc-56f6bd887d-smpvf"]
Nov 03 22:22:23.619227 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.619135    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"]
Nov 03 22:22:23.625838 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:23.625803    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e15bbc-667cbc4dc-2nf7c"]
Nov 03 22:22:25.135046 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:25.135014    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221a1e8e-d593-4e40-b27a-b201d76c7e46" path="/var/lib/kubelet/pods/221a1e8e-d593-4e40-b27a-b201d76c7e46/volumes"
Nov 03 22:22:25.135470 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:25.135400    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81d8fbf-b920-45cd-94b7-d7f97aebaa97" path="/var/lib/kubelet/pods/c81d8fbf-b920-45cd-94b7-d7f97aebaa97/volumes"
Nov 03 22:22:26.807130 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:26.807094    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"]
Nov 03 22:22:26.857229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:26.857173    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"]
Nov 03 22:22:26.858603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:26.858573    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"]
Nov 03 22:22:29.325118 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.325085    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"]
Nov 03 22:22:29.325552 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.325316    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" podUID="dd9bb00a-9e87-4091-957e-a09f10c147e0" containerName="maas-api" containerID="cri-o://4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d" gracePeriod=30
Nov 03 22:22:29.610093 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.610057    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 22:22:29.762045 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.762010    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\") pod \"dd9bb00a-9e87-4091-957e-a09f10c147e0\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") "
Nov 03 22:22:29.762045 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.762049    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm\" (UniqueName: \"kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm\") pod \"dd9bb00a-9e87-4091-957e-a09f10c147e0\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") "
Nov 03 22:22:29.762315 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.762075    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs\") pod \"dd9bb00a-9e87-4091-957e-a09f10c147e0\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") "
Nov 03 22:22:29.762315 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.762113    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca\") pod \"dd9bb00a-9e87-4091-957e-a09f10c147e0\" (UID: \"dd9bb00a-9e87-4091-957e-a09f10c147e0\") "
Nov 03 22:22:29.762579 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.762557    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca" pod "dd9bb00a-9e87-4091-957e-a09f10c147e0" (UID: "dd9bb00a-9e87-4091-957e-a09f10c147e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:29.764495 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.764466    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls" pod "dd9bb00a-9e87-4091-957e-a09f10c147e0" (UID: "dd9bb00a-9e87-4091-957e-a09f10c147e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:29.764597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.764470    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm" pod "dd9bb00a-9e87-4091-957e-a09f10c147e0" (UID: "dd9bb00a-9e87-4091-957e-a09f10c147e0"). InnerVolumeSpecName "kube-api-access-zpjzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:29.764597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.764548    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs" pod "dd9bb00a-9e87-4091-957e-a09f10c147e0" (UID: "dd9bb00a-9e87-4091-957e-a09f10c147e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:29.863063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.862974    5244 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9bb00a-9e87-4091-957e-a09f10c147e0-openshift-service-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:29.863063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.863007    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-maas-api-tls\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:29.863063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.863019    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjzm\" (UniqueName: \"kubernetes.io/projected/dd9bb00a-9e87-4091-957e-a09f10c147e0-kube-api-access-zpjzm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:29.863063 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:29.863028    5244 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9bb00a-9e87-4091-957e-a09f10c147e0-metrics-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:30.392828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.392786    5244 generic.go:356] "Generic (PLEG): container finished" podID="dd9bb00a-9e87-4091-957e-a09f10c147e0" containerID="4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d" exitCode=0
Nov 03 22:22:30.392828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.392829    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"
Nov 03 22:22:30.393403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.392869    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" event={"ID":"dd9bb00a-9e87-4091-957e-a09f10c147e0","Type":"ContainerDied","Data":"4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d"}
Nov 03 22:22:30.393403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.392929    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r" event={"ID":"dd9bb00a-9e87-4091-957e-a09f10c147e0","Type":"ContainerDied","Data":"7c533164224de795669e4562c6c4a82c06f5f6c1fe2e51c5671e6deb384ae310"}
Nov 03 22:22:30.393403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.392956    5244 scope.go:122] "RemoveContainer" containerID="4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d"
Nov 03 22:22:30.402839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.402816    5244 scope.go:122] "RemoveContainer" containerID="4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d"
Nov 03 22:22:30.403122 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:30.403100    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d\": container with ID starting with 4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d not found: ID does not exist" containerID="4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d"
Nov 03 22:22:30.403198 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.403135    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d"} err="failed to get container status \"4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d\": rpc error: code = NotFound desc = could not find container \"4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d\": container with ID starting with 4bd2cacd0a39b3ee39082cbef4df773442bdacd32b363d75feba0a9306d7bc4d not found: ID does not exist"
Nov 03 22:22:30.555493 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.555457    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"]
Nov 03 22:22:30.604785 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:30.604751    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f463c9-7679dd8f88-7895r"]
Nov 03 22:22:31.136896 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.136863    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9bb00a-9e87-4091-957e-a09f10c147e0" path="/var/lib/kubelet/pods/dd9bb00a-9e87-4091-957e-a09f10c147e0/volumes"
Nov 03 22:22:31.610048 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.610016    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"]
Nov 03 22:22:31.659301 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.659268    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f463c9-688vk"]
Nov 03 22:22:31.932794 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.932763    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"]
Nov 03 22:22:31.933145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933133    5244 cpu_manager.go:414] "containerMap: removing container" podUID="221a1e8e-d593-4e40-b27a-b201d76c7e46" containerName="payload-pre-processing"
Nov 03 22:22:31.933212 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933146    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221a1e8e-d593-4e40-b27a-b201d76c7e46" containerName="payload-pre-processing"
Nov 03 22:22:31.933212 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933172    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c81d8fbf-b920-45cd-94b7-d7f97aebaa97" containerName="payload-processing"
Nov 03 22:22:31.933212 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933184    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81d8fbf-b920-45cd-94b7-d7f97aebaa97" containerName="payload-processing"
Nov 03 22:22:31.933212 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933194    5244 cpu_manager.go:414] "containerMap: removing container" podUID="86cf5517-0463-4d69-9384-f69c91e7505f" containerName="revoke-keys"
Nov 03 22:22:31.933212 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933199    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86cf5517-0463-4d69-9384-f69c91e7505f" containerName="revoke-keys"
Nov 03 22:22:31.933212 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933213    5244 cpu_manager.go:414] "containerMap: removing container" podUID="dd9bb00a-9e87-4091-957e-a09f10c147e0" containerName="maas-api"
Nov 03 22:22:31.933450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933217    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9bb00a-9e87-4091-957e-a09f10c147e0" containerName="maas-api"
Nov 03 22:22:31.933450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933268    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81d8fbf-b920-45cd-94b7-d7f97aebaa97" containerName="payload-processing"
Nov 03 22:22:31.933450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933276    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="86cf5517-0463-4d69-9384-f69c91e7505f" containerName="revoke-keys"
Nov 03 22:22:31.933450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933284    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="221a1e8e-d593-4e40-b27a-b201d76c7e46" containerName="payload-pre-processing"
Nov 03 22:22:31.933450 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.933292    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9bb00a-9e87-4091-957e-a09f10c147e0" containerName="maas-api"
Nov 03 22:22:31.938177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.938155    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:31.993721 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:31.993686    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"]
Nov 03 22:22:32.040990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.040962    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:32.041185 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.041081    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:32.087089 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.087053    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.087287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.087196    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.087287 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.087258    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6f4n\" (UniqueName: \"kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.188612 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.188501    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6f4n\" (UniqueName: \"kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.188612 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.188571    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.188891 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.188644    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.189229 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.189210    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.191175 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.191150    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.245224 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.245189    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6f4n\" (UniqueName: \"kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n\") pod \"maas-api-revoke-sub-e4063bf5-gpkbq\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.251026 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.250986    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:32.623699 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.623600    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"]
Nov 03 22:22:32.626661 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:22:32.626630    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfa5197_b69d_411d_a8b7_01d4ebad5407.slice/crio-54fb570db44c490916b777e827e7fd6dc367f672eb9ad7d4e8d0107e0b11bf6d WatchSource:0}: Error finding container 54fb570db44c490916b777e827e7fd6dc367f672eb9ad7d4e8d0107e0b11bf6d: Status 404 returned error can't find the container with id 54fb570db44c490916b777e827e7fd6dc367f672eb9ad7d4e8d0107e0b11bf6d
Nov 03 22:22:32.973344 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.973301    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"]
Nov 03 22:22:32.975357 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:32.975313    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ddbba-g4h6k"]
Nov 03 22:22:33.135924 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:33.135886    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d" path="/var/lib/kubelet/pods/1b8c2d2d-0a5f-4984-b5d0-27d3ccd14a9d/volumes"
Nov 03 22:22:33.136291 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:33.136277    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86cf5517-0463-4d69-9384-f69c91e7505f" path="/var/lib/kubelet/pods/86cf5517-0463-4d69-9384-f69c91e7505f/volumes"
Nov 03 22:22:33.409637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:33.409544    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq" event={"ID":"4bfa5197-b69d-411d-a8b7-01d4ebad5407","Type":"ContainerStarted","Data":"167fdb8d031162a0098ef7febcc413856210f73c20673944a3cec0931c864ac5"}
Nov 03 22:22:33.409637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:33.409595    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq" event={"ID":"4bfa5197-b69d-411d-a8b7-01d4ebad5407","Type":"ContainerStarted","Data":"54fb570db44c490916b777e827e7fd6dc367f672eb9ad7d4e8d0107e0b11bf6d"}
Nov 03 22:22:33.523117 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:33.523072    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq" podStartSLOduration=2.523058229 podStartE2EDuration="2.523058229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:33.522670396 +0000 UTC m=+4024.506666966" watchObservedRunningTime="2026-11-03 22:22:33.523058229 +0000 UTC m=+4024.507054818"
Nov 03 22:22:34.416512 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:34.416478    5244 generic.go:356] "Generic (PLEG): container finished" podID="4bfa5197-b69d-411d-a8b7-01d4ebad5407" containerID="167fdb8d031162a0098ef7febcc413856210f73c20673944a3cec0931c864ac5" exitCode=0
Nov 03 22:22:34.416999 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:34.416585    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq" event={"ID":"4bfa5197-b69d-411d-a8b7-01d4ebad5407","Type":"ContainerDied","Data":"167fdb8d031162a0098ef7febcc413856210f73c20673944a3cec0931c864ac5"}
Nov 03 22:22:35.566136 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.566108    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:35.727881 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.727840    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca\") pod \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") "
Nov 03 22:22:35.728101 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.727913    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca\") pod \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") "
Nov 03 22:22:35.728101 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.727971    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n\" (UniqueName: \"kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n\") pod \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\" (UID: \"4bfa5197-b69d-411d-a8b7-01d4ebad5407\") "
Nov 03 22:22:35.728297 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.728273    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca" pod "4bfa5197-b69d-411d-a8b7-01d4ebad5407" (UID: "4bfa5197-b69d-411d-a8b7-01d4ebad5407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:35.730176 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.730154    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca" pod "4bfa5197-b69d-411d-a8b7-01d4ebad5407" (UID: "4bfa5197-b69d-411d-a8b7-01d4ebad5407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:35.730277 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.730213    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n" pod "4bfa5197-b69d-411d-a8b7-01d4ebad5407" (UID: "4bfa5197-b69d-411d-a8b7-01d4ebad5407"). InnerVolumeSpecName "kube-api-access-g6f4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:35.829420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.829390    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:35.829420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.829415    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bfa5197-b69d-411d-a8b7-01d4ebad5407-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:35.829420 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:35.829424    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6f4n\" (UniqueName: \"kubernetes.io/projected/4bfa5197-b69d-411d-a8b7-01d4ebad5407-kube-api-access-g6f4n\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:36.435935 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:36.435898    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq"
Nov 03 22:22:36.436125 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:36.435930    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4063bf5-gpkbq" event={"ID":"4bfa5197-b69d-411d-a8b7-01d4ebad5407","Type":"ContainerDied","Data":"54fb570db44c490916b777e827e7fd6dc367f672eb9ad7d4e8d0107e0b11bf6d"}
Nov 03 22:22:36.436125 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:36.436018    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54fb570db44c490916b777e827e7fd6dc367f672eb9ad7d4e8d0107e0b11bf6d"
Nov 03 22:22:41.827922 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:41.827875    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7" podUID="83390212-04c2-49ff-824d-8cb84bb7c9ac" containerName="main" containerID="cri-o://934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35" gracePeriod=45
Nov 03 22:22:41.877764 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:41.877709    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main" containerID="cri-o://f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f" gracePeriod=45
Nov 03 22:22:41.877919 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:41.877875    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main" containerID="cri-o://cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4" gracePeriod=45
Nov 03 22:22:41.901584 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:41.901551    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Nov 03 22:22:42.156977 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.156952    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 22:22:42.186994 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.186969    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 22:22:42.191057 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.191037    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 22:22:42.293785 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.293753    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home\") pod \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") "
Nov 03 22:22:42.293988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.293808    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache\") pod \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") "
Nov 03 22:22:42.293988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.293843    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt\") pod \"83390212-04c2-49ff-824d-8cb84bb7c9ac\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") "
Nov 03 22:22:42.293988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.293873    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm\") pod \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") "
Nov 03 22:22:42.293988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.293902    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs\") pod \"83390212-04c2-49ff-824d-8cb84bb7c9ac\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") "
Nov 03 22:22:42.293988 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.293930    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv\" (UniqueName: \"kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv\") pod \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") "
Nov 03 22:22:42.294279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294092    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir\") pod \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") "
Nov 03 22:22:42.294279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294105    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache" pod "4fd6e820-8ece-4b96-b89b-82643f7dd79f" (UID: "4fd6e820-8ece-4b96-b89b-82643f7dd79f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.294279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294133    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir\") pod \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") "
Nov 03 22:22:42.294279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294186    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k\" (UniqueName: \"kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k\") pod \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") "
Nov 03 22:22:42.294279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294226    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs\") pod \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") "
Nov 03 22:22:42.294279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294253    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm\") pod \"83390212-04c2-49ff-824d-8cb84bb7c9ac\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294285    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home\") pod \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294322    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir" pod "44ef4c6b-0630-4abb-b04b-0d2e455d8b28" (UID: "44ef4c6b-0630-4abb-b04b-0d2e455d8b28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294352    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache\") pod \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294378    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home" pod "44ef4c6b-0630-4abb-b04b-0d2e455d8b28" (UID: "44ef4c6b-0630-4abb-b04b-0d2e455d8b28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294382    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs\" (UniqueName: \"kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs\") pod \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\" (UID: \"44ef4c6b-0630-4abb-b04b-0d2e455d8b28\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294456    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm\") pod \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\" (UID: \"4fd6e820-8ece-4b96-b89b-82643f7dd79f\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294502    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home\") pod \"83390212-04c2-49ff-824d-8cb84bb7c9ac\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294540    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache\") pod \"83390212-04c2-49ff-824d-8cb84bb7c9ac\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") "
Nov 03 22:22:42.294637 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.294621    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir\") pod \"83390212-04c2-49ff-824d-8cb84bb7c9ac\" (UID: \"83390212-04c2-49ff-824d-8cb84bb7c9ac\") "
Nov 03 22:22:42.295188 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295058    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir" pod "4fd6e820-8ece-4b96-b89b-82643f7dd79f" (UID: "4fd6e820-8ece-4b96-b89b-82643f7dd79f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.295283 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295260    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir" pod "83390212-04c2-49ff-824d-8cb84bb7c9ac" (UID: "83390212-04c2-49ff-824d-8cb84bb7c9ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.295396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295307    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.295396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295361    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.295396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295378    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.295396 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295392    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.295621 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295583    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home" pod "4fd6e820-8ece-4b96-b89b-82643f7dd79f" (UID: "4fd6e820-8ece-4b96-b89b-82643f7dd79f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.296027 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.295924    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home" pod "83390212-04c2-49ff-824d-8cb84bb7c9ac" (UID: "83390212-04c2-49ff-824d-8cb84bb7c9ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.296305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.296270    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache" pod "83390212-04c2-49ff-824d-8cb84bb7c9ac" (UID: "83390212-04c2-49ff-824d-8cb84bb7c9ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.296305 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.296288    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache" pod "44ef4c6b-0630-4abb-b04b-0d2e455d8b28" (UID: "44ef4c6b-0630-4abb-b04b-0d2e455d8b28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.297843 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.297804    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm" pod "4fd6e820-8ece-4b96-b89b-82643f7dd79f" (UID: "4fd6e820-8ece-4b96-b89b-82643f7dd79f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.298304 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.298206    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs" pod "4fd6e820-8ece-4b96-b89b-82643f7dd79f" (UID: "4fd6e820-8ece-4b96-b89b-82643f7dd79f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:42.298304 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.298277    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs" pod "44ef4c6b-0630-4abb-b04b-0d2e455d8b28" (UID: "44ef4c6b-0630-4abb-b04b-0d2e455d8b28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:42.298584 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.298543    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs" pod "83390212-04c2-49ff-824d-8cb84bb7c9ac" (UID: "83390212-04c2-49ff-824d-8cb84bb7c9ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:42.298684 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.298630    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt" pod "83390212-04c2-49ff-824d-8cb84bb7c9ac" (UID: "83390212-04c2-49ff-824d-8cb84bb7c9ac"). InnerVolumeSpecName "kube-api-access-chjvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:42.298684 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.298659    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv" pod "4fd6e820-8ece-4b96-b89b-82643f7dd79f" (UID: "4fd6e820-8ece-4b96-b89b-82643f7dd79f"). InnerVolumeSpecName "kube-api-access-bmjpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:42.298770 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.298703    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k" pod "44ef4c6b-0630-4abb-b04b-0d2e455d8b28" (UID: "44ef4c6b-0630-4abb-b04b-0d2e455d8b28"). InnerVolumeSpecName "kube-api-access-7l29k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:42.299603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.299583    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm" pod "44ef4c6b-0630-4abb-b04b-0d2e455d8b28" (UID: "44ef4c6b-0630-4abb-b04b-0d2e455d8b28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.300186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.300165    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm" pod "83390212-04c2-49ff-824d-8cb84bb7c9ac" (UID: "83390212-04c2-49ff-824d-8cb84bb7c9ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396174    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chjvt\" (UniqueName: \"kubernetes.io/projected/83390212-04c2-49ff-824d-8cb84bb7c9ac-kube-api-access-chjvt\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396206    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396215    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83390212-04c2-49ff-824d-8cb84bb7c9ac-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396224    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmjpv\" (UniqueName: \"kubernetes.io/projected/4fd6e820-8ece-4b96-b89b-82643f7dd79f-kube-api-access-bmjpv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396232    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l29k\" (UniqueName: \"kubernetes.io/projected/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-kube-api-access-7l29k\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396244    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd6e820-8ece-4b96-b89b-82643f7dd79f-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396256    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396268    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396279    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396291    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ef4c6b-0630-4abb-b04b-0d2e455d8b28-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396303    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd6e820-8ece-4b96-b89b-82643f7dd79f-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396313    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396347    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.396664 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.396364    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83390212-04c2-49ff-824d-8cb84bb7c9ac-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:42.474116 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.474085    5244 generic.go:356] "Generic (PLEG): container finished" podID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerID="cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4" exitCode=0
Nov 03 22:22:42.474388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.474153    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"
Nov 03 22:22:42.474388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.474166    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" event={"ID":"4fd6e820-8ece-4b96-b89b-82643f7dd79f","Type":"ContainerDied","Data":"cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4"}
Nov 03 22:22:42.474388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.474218    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj" event={"ID":"4fd6e820-8ece-4b96-b89b-82643f7dd79f","Type":"ContainerDied","Data":"a123cc1fead60b1af999a1fcdd1ab345a3aab10bf6776078ab0ea13241e8af47"}
Nov 03 22:22:42.474388 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.474242    5244 scope.go:122] "RemoveContainer" containerID="cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4"
Nov 03 22:22:42.475639 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.475616    5244 generic.go:356] "Generic (PLEG): container finished" podID="83390212-04c2-49ff-824d-8cb84bb7c9ac" containerID="934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35" exitCode=0
Nov 03 22:22:42.475765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.475641    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7" event={"ID":"83390212-04c2-49ff-824d-8cb84bb7c9ac","Type":"ContainerDied","Data":"934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35"}
Nov 03 22:22:42.475765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.475674    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"
Nov 03 22:22:42.475765 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.475682    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7" event={"ID":"83390212-04c2-49ff-824d-8cb84bb7c9ac","Type":"ContainerDied","Data":"544deba14b30bf8f15c2ff5f143313c50ed92e856191395647d48072a79f4157"}
Nov 03 22:22:42.477543 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.477522    5244 generic.go:356] "Generic (PLEG): container finished" podID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerID="f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f" exitCode=0
Nov 03 22:22:42.477647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.477561    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" event={"ID":"44ef4c6b-0630-4abb-b04b-0d2e455d8b28","Type":"ContainerDied","Data":"f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f"}
Nov 03 22:22:42.477647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.477587    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8" event={"ID":"44ef4c6b-0630-4abb-b04b-0d2e455d8b28","Type":"ContainerDied","Data":"4c2745f34dcc4429aff36560a903b7bd7776f246ce1f121d4fbe74592313c023"}
Nov 03 22:22:42.477647 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.477590    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"
Nov 03 22:22:42.484708 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.484686    5244 scope.go:122] "RemoveContainer" containerID="cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4"
Nov 03 22:22:42.485016 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:42.484995    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4\": container with ID starting with cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4 not found: ID does not exist" containerID="cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4"
Nov 03 22:22:42.485129 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.485024    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4"} err="failed to get container status \"cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4\": rpc error: code = NotFound desc = could not find container \"cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4\": container with ID starting with cdd5a5429d66c700f8b4b2583e4025d0e6df9ebe72e9ca02319d5e23ea02eab4 not found: ID does not exist"
Nov 03 22:22:42.485129 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.485046    5244 scope.go:122] "RemoveContainer" containerID="934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35"
Nov 03 22:22:42.496634 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.496611    5244 scope.go:122] "RemoveContainer" containerID="934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35"
Nov 03 22:22:42.497143 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:42.497007    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35\": container with ID starting with 934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35 not found: ID does not exist" containerID="934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35"
Nov 03 22:22:42.497143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.497042    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35"} err="failed to get container status \"934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35\": rpc error: code = NotFound desc = could not find container \"934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35\": container with ID starting with 934bebc3b554776a00c86b3ec68a0a7617857e19fa4d40ab20bf69e636110f35 not found: ID does not exist"
Nov 03 22:22:42.497143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.497064    5244 scope.go:122] "RemoveContainer" containerID="f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f"
Nov 03 22:22:42.507695 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.507672    5244 scope.go:122] "RemoveContainer" containerID="f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f"
Nov 03 22:22:42.508048 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:42.508018    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f\": container with ID starting with f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f not found: ID does not exist" containerID="f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f"
Nov 03 22:22:42.508145 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.508061    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f"} err="failed to get container status \"f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f\": rpc error: code = NotFound desc = could not find container \"f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f\": container with ID starting with f14981fe156a79834bf3968198e25b80d1bece6b55ebec3ddf71f1a34e0d279f not found: ID does not exist"
Nov 03 22:22:42.634230 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.634192    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"]
Nov 03 22:22:42.638010 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.637980    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-2-w0-7ee809-kserve-6d774f65f6-q6gg7"]
Nov 03 22:22:42.745440 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.745407    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"]
Nov 03 22:22:42.748729 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.748691    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/distinct-w0-7ee809-kserve-647947879f-ft7j8"]
Nov 03 22:22:42.857467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.857425    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"]
Nov 03 22:22:42.861034 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:42.861005    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7ee809/simulator-w0-7ee809-kserve-6b6d545c85-dd4lj"]
Nov 03 22:22:43.135534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:43.135453    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" path="/var/lib/kubelet/pods/44ef4c6b-0630-4abb-b04b-0d2e455d8b28/volumes"
Nov 03 22:22:43.135866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:43.135853    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" path="/var/lib/kubelet/pods/4fd6e820-8ece-4b96-b89b-82643f7dd79f/volumes"
Nov 03 22:22:43.136202 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:43.136190    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83390212-04c2-49ff-824d-8cb84bb7c9ac" path="/var/lib/kubelet/pods/83390212-04c2-49ff-824d-8cb84bb7c9ac/volumes"
Nov 03 22:22:45.352826 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:45.352791    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"]
Nov 03 22:22:45.402196 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:45.402156    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"]
Nov 03 22:22:45.403641 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:45.403615    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"]
Nov 03 22:22:52.436715 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:22:52.436672    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:22:54.416754 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.416716    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"]
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417096    5244 cpu_manager.go:414] "containerMap: removing container" podUID="83390212-04c2-49ff-824d-8cb84bb7c9ac" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417108    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83390212-04c2-49ff-824d-8cb84bb7c9ac" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417148    5244 cpu_manager.go:414] "containerMap: removing container" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417154    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417163    5244 cpu_manager.go:414] "containerMap: removing container" podUID="4bfa5197-b69d-411d-a8b7-01d4ebad5407" containerName="revoke-keys"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417168    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfa5197-b69d-411d-a8b7-01d4ebad5407" containerName="revoke-keys"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417177    5244 cpu_manager.go:414] "containerMap: removing container" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417192    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417255    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd6e820-8ece-4b96-b89b-82643f7dd79f" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417266    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ef4c6b-0630-4abb-b04b-0d2e455d8b28" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417274    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="83390212-04c2-49ff-824d-8cb84bb7c9ac" containerName="main"
Nov 03 22:22:54.417300 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.417284    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfa5197-b69d-411d-a8b7-01d4ebad5407" containerName="revoke-keys"
Nov 03 22:22:54.424947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.424853    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.428650 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.428625    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:22:54.480411 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.480376    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvpbj\""
Nov 03 22:22:54.488123 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.488097    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"]
Nov 03 22:22:54.509141 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.509108    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk292\" (UniqueName: \"kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.509309 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.509291    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.509531 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.509507    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.553062 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.553028    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 22:22:54.610571 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.610531    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk292\" (UniqueName: \"kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.610808 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.610590    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.610808 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.610651    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.611535 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.611503    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.613200 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.613164    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.667217 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.667143    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk292\" (UniqueName: \"kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292\") pod \"maas-api-revoke-sub-78b527eb-fqdg6\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:54.741288 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:54.741251    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:55.162950 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:55.162920    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"]
Nov 03 22:22:55.166443 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:22:55.166413    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8956965_8df9_45f5_b88a_3ff623369d4d.slice/crio-d83cc3d8bb8566e8ed452bed0e3feb154175536205da62af41d3ccd345fd8475 WatchSource:0}: Error finding container d83cc3d8bb8566e8ed452bed0e3feb154175536205da62af41d3ccd345fd8475: Status 404 returned error can't find the container with id d83cc3d8bb8566e8ed452bed0e3feb154175536205da62af41d3ccd345fd8475
Nov 03 22:22:55.573356 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:55.573086    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6" event={"ID":"f8956965-8df9-45f5-b88a-3ff623369d4d","Type":"ContainerStarted","Data":"4d734f8dc008bdb35b462988ed9fea59364b76a1f1e70230d15aaa369c7aac2d"}
Nov 03 22:22:55.573356 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:55.573160    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6" event={"ID":"f8956965-8df9-45f5-b88a-3ff623369d4d","Type":"ContainerStarted","Data":"d83cc3d8bb8566e8ed452bed0e3feb154175536205da62af41d3ccd345fd8475"}
Nov 03 22:22:55.688019 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:55.687945    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6" podStartSLOduration=1.6879235910000001 podStartE2EDuration="1.687923591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:55.686259483 +0000 UTC m=+4046.670256063" watchObservedRunningTime="2026-11-03 22:22:55.687923591 +0000 UTC m=+4046.671920151"
Nov 03 22:22:56.579981 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:56.579877    5244 generic.go:356] "Generic (PLEG): container finished" podID="f8956965-8df9-45f5-b88a-3ff623369d4d" containerID="4d734f8dc008bdb35b462988ed9fea59364b76a1f1e70230d15aaa369c7aac2d" exitCode=0
Nov 03 22:22:56.579981 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:56.579931    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6" event={"ID":"f8956965-8df9-45f5-b88a-3ff623369d4d","Type":"ContainerDied","Data":"4d734f8dc008bdb35b462988ed9fea59364b76a1f1e70230d15aaa369c7aac2d"}
Nov 03 22:22:57.707017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.706993    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:57.743958 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.743927    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca\") pod \"f8956965-8df9-45f5-b88a-3ff623369d4d\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") "
Nov 03 22:22:57.744098 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.743966    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292\" (UniqueName: \"kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292\") pod \"f8956965-8df9-45f5-b88a-3ff623369d4d\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") "
Nov 03 22:22:57.744098 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.743988    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca\") pod \"f8956965-8df9-45f5-b88a-3ff623369d4d\" (UID: \"f8956965-8df9-45f5-b88a-3ff623369d4d\") "
Nov 03 22:22:57.744423 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.744395    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca" pod "f8956965-8df9-45f5-b88a-3ff623369d4d" (UID: "f8956965-8df9-45f5-b88a-3ff623369d4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:22:57.746102 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.746078    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca" pod "f8956965-8df9-45f5-b88a-3ff623369d4d" (UID: "f8956965-8df9-45f5-b88a-3ff623369d4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:22:57.746193 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.746160    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292" pod "f8956965-8df9-45f5-b88a-3ff623369d4d" (UID: "f8956965-8df9-45f5-b88a-3ff623369d4d"). InnerVolumeSpecName "kube-api-access-zk292". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:22:57.845709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.845622    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-xks-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:57.845709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.845654    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk292\" (UniqueName: \"kubernetes.io/projected/f8956965-8df9-45f5-b88a-3ff623369d4d-kube-api-access-zk292\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:57.845709 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:57.845665    5244 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8956965-8df9-45f5-b88a-3ff623369d4d-maas-api-openshift-ca\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:22:58.592759 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:58.592727    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6"
Nov 03 22:22:58.592939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:58.592759    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78b527eb-fqdg6" event={"ID":"f8956965-8df9-45f5-b88a-3ff623369d4d","Type":"ContainerDied","Data":"d83cc3d8bb8566e8ed452bed0e3feb154175536205da62af41d3ccd345fd8475"}
Nov 03 22:22:58.592939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:22:58.592813    5244 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d83cc3d8bb8566e8ed452bed0e3feb154175536205da62af41d3ccd345fd8475"
Nov 03 22:23:00.373782 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.373734    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" podUID="715ce125-f53c-4b2e-848e-029866cbc5cf" containerName="main" containerID="cri-o://0de62eb2a43f31ac9cdaec25c7027cbaa10b801ccfbc6721510b2b9793ec379f" gracePeriod=45
Nov 03 22:23:00.422091 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.422056    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main" containerID="cri-o://238123f07649f7d08ff0efcb54f6cf8aeaf9342cf57ac95f9edfd01183045a8d" gracePeriod=45
Nov 03 22:23:00.422609 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.422582    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main" containerID="cri-o://20bb461da74e86996bb5988499c52b6bd1fa9fe6c596e214e4e1a018e35226f5" gracePeriod=45
Nov 03 22:23:00.517804 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.517756    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": dial tcp 10.133.0.33:8000: connect: connection refused"
Nov 03 22:23:00.608997 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.608956    5244 generic.go:356] "Generic (PLEG): container finished" podID="715ce125-f53c-4b2e-848e-029866cbc5cf" containerID="0de62eb2a43f31ac9cdaec25c7027cbaa10b801ccfbc6721510b2b9793ec379f" exitCode=0
Nov 03 22:23:00.609177 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.609074    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" event={"ID":"715ce125-f53c-4b2e-848e-029866cbc5cf","Type":"ContainerDied","Data":"0de62eb2a43f31ac9cdaec25c7027cbaa10b801ccfbc6721510b2b9793ec379f"}
Nov 03 22:23:00.610989 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.610958    5244 generic.go:356] "Generic (PLEG): container finished" podID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerID="20bb461da74e86996bb5988499c52b6bd1fa9fe6c596e214e4e1a018e35226f5" exitCode=0
Nov 03 22:23:00.611137 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.611088    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" event={"ID":"70773fa8-3c22-4f5f-8086-cc786e65c043","Type":"ContainerDied","Data":"20bb461da74e86996bb5988499c52b6bd1fa9fe6c596e214e4e1a018e35226f5"}
Nov 03 22:23:00.612614 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.612591    5244 generic.go:356] "Generic (PLEG): container finished" podID="136a8937-a0c9-4e97-bf99-f4e708359238" containerID="238123f07649f7d08ff0efcb54f6cf8aeaf9342cf57ac95f9edfd01183045a8d" exitCode=0
Nov 03 22:23:00.612807 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.612643    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" event={"ID":"136a8937-a0c9-4e97-bf99-f4e708359238","Type":"ContainerDied","Data":"238123f07649f7d08ff0efcb54f6cf8aeaf9342cf57ac95f9edfd01183045a8d"}
Nov 03 22:23:00.713121 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.712003    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 22:23:00.735003 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.734976    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 22:23:00.738010 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.737990    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 22:23:00.777120 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777089    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home\") pod \"715ce125-f53c-4b2e-848e-029866cbc5cf\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") "
Nov 03 22:23:00.777377 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777139    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache\") pod \"715ce125-f53c-4b2e-848e-029866cbc5cf\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") "
Nov 03 22:23:00.777377 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777170    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs\") pod \"715ce125-f53c-4b2e-848e-029866cbc5cf\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") "
Nov 03 22:23:00.777377 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777236    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home\") pod \"70773fa8-3c22-4f5f-8086-cc786e65c043\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") "
Nov 03 22:23:00.777377 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777263    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs\" (UniqueName: \"kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs\") pod \"70773fa8-3c22-4f5f-8086-cc786e65c043\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") "
Nov 03 22:23:00.777620 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777484    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home" pod "715ce125-f53c-4b2e-848e-029866cbc5cf" (UID: "715ce125-f53c-4b2e-848e-029866cbc5cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.777620 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777589    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache\") pod \"136a8937-a0c9-4e97-bf99-f4e708359238\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") "
Nov 03 22:23:00.777735 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777613    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache" pod "715ce125-f53c-4b2e-848e-029866cbc5cf" (UID: "715ce125-f53c-4b2e-848e-029866cbc5cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.777735 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777649    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir\") pod \"136a8937-a0c9-4e97-bf99-f4e708359238\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") "
Nov 03 22:23:00.777735 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777680    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache\") pod \"70773fa8-3c22-4f5f-8086-cc786e65c043\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") "
Nov 03 22:23:00.777735 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777708    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm\") pod \"136a8937-a0c9-4e97-bf99-f4e708359238\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") "
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777770    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm\") pod \"70773fa8-3c22-4f5f-8086-cc786e65c043\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") "
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777800    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir\") pod \"715ce125-f53c-4b2e-848e-029866cbc5cf\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") "
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777832    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv\" (UniqueName: \"kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv\") pod \"715ce125-f53c-4b2e-848e-029866cbc5cf\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") "
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777835    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache" pod "136a8937-a0c9-4e97-bf99-f4e708359238" (UID: "136a8937-a0c9-4e97-bf99-f4e708359238"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777881    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir\") pod \"70773fa8-3c22-4f5f-8086-cc786e65c043\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") "
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777905    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs\" (UniqueName: \"kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs\") pod \"136a8937-a0c9-4e97-bf99-f4e708359238\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") "
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777919    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache" pod "70773fa8-3c22-4f5f-8086-cc786e65c043" (UID: "70773fa8-3c22-4f5f-8086-cc786e65c043"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.777963 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777936    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6\" (UniqueName: \"kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6\") pod \"136a8937-a0c9-4e97-bf99-f4e708359238\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") "
Nov 03 22:23:00.778390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.777980    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home\") pod \"136a8937-a0c9-4e97-bf99-f4e708359238\" (UID: \"136a8937-a0c9-4e97-bf99-f4e708359238\") "
Nov 03 22:23:00.778390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778047    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm\") pod \"715ce125-f53c-4b2e-848e-029866cbc5cf\" (UID: \"715ce125-f53c-4b2e-848e-029866cbc5cf\") "
Nov 03 22:23:00.778390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778047    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir" pod "715ce125-f53c-4b2e-848e-029866cbc5cf" (UID: "715ce125-f53c-4b2e-848e-029866cbc5cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.778390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778087    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv\" (UniqueName: \"kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv\") pod \"70773fa8-3c22-4f5f-8086-cc786e65c043\" (UID: \"70773fa8-3c22-4f5f-8086-cc786e65c043\") "
Nov 03 22:23:00.778390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778173    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir" pod "136a8937-a0c9-4e97-bf99-f4e708359238" (UID: "136a8937-a0c9-4e97-bf99-f4e708359238"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.778390 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778247    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir" pod "70773fa8-3c22-4f5f-8086-cc786e65c043" (UID: "70773fa8-3c22-4f5f-8086-cc786e65c043"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.778748 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778533    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home" pod "136a8937-a0c9-4e97-bf99-f4e708359238" (UID: "136a8937-a0c9-4e97-bf99-f4e708359238"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.778905    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.779229    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.779247    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.779275    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.779288    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.779301    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.779363 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.779312    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.780037 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.780002    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home" pod "70773fa8-3c22-4f5f-8086-cc786e65c043" (UID: "70773fa8-3c22-4f5f-8086-cc786e65c043"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.780779 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.780741    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm" pod "70773fa8-3c22-4f5f-8086-cc786e65c043" (UID: "70773fa8-3c22-4f5f-8086-cc786e65c043"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.781722 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.781697    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs" pod "136a8937-a0c9-4e97-bf99-f4e708359238" (UID: "136a8937-a0c9-4e97-bf99-f4e708359238"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:00.781954 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.781931    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs" pod "715ce125-f53c-4b2e-848e-029866cbc5cf" (UID: "715ce125-f53c-4b2e-848e-029866cbc5cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:00.782076 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.781958    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs" pod "70773fa8-3c22-4f5f-8086-cc786e65c043" (UID: "70773fa8-3c22-4f5f-8086-cc786e65c043"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:00.782076 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.782022    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv" pod "715ce125-f53c-4b2e-848e-029866cbc5cf" (UID: "715ce125-f53c-4b2e-848e-029866cbc5cf"). InnerVolumeSpecName "kube-api-access-xvdpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:00.782523 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.782498    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv" pod "70773fa8-3c22-4f5f-8086-cc786e65c043" (UID: "70773fa8-3c22-4f5f-8086-cc786e65c043"). InnerVolumeSpecName "kube-api-access-tgqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:00.782747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.782719    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6" pod "136a8937-a0c9-4e97-bf99-f4e708359238" (UID: "136a8937-a0c9-4e97-bf99-f4e708359238"). InnerVolumeSpecName "kube-api-access-fkzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:00.783508 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.783485    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm" pod "136a8937-a0c9-4e97-bf99-f4e708359238" (UID: "136a8937-a0c9-4e97-bf99-f4e708359238"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.784623 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.784603    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm" pod "715ce125-f53c-4b2e-848e-029866cbc5cf" (UID: "715ce125-f53c-4b2e-848e-029866cbc5cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.879915    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.879943    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70773fa8-3c22-4f5f-8086-cc786e65c043-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.879955    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.879965    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70773fa8-3c22-4f5f-8086-cc786e65c043-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.879979    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvdpv\" (UniqueName: \"kubernetes.io/projected/715ce125-f53c-4b2e-848e-029866cbc5cf-kube-api-access-xvdpv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.879993    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136a8937-a0c9-4e97-bf99-f4e708359238-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.880005    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkzt6\" (UniqueName: \"kubernetes.io/projected/136a8937-a0c9-4e97-bf99-f4e708359238-kube-api-access-fkzt6\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.880017    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136a8937-a0c9-4e97-bf99-f4e708359238-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.880029    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/715ce125-f53c-4b2e-848e-029866cbc5cf-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.880040    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgqgv\" (UniqueName: \"kubernetes.io/projected/70773fa8-3c22-4f5f-8086-cc786e65c043-kube-api-access-tgqgv\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:00.880433 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:00.880051    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/715ce125-f53c-4b2e-848e-029866cbc5cf-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:01.620125 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.620096    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"
Nov 03 22:23:01.620656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.620136    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj" event={"ID":"136a8937-a0c9-4e97-bf99-f4e708359238","Type":"ContainerDied","Data":"83141846d63cb06bfbf6a91e9e23df43e4463407ccef4c56cfff5fadc8c167d1"}
Nov 03 22:23:01.620656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.620186    5244 scope.go:122] "RemoveContainer" containerID="238123f07649f7d08ff0efcb54f6cf8aeaf9342cf57ac95f9edfd01183045a8d"
Nov 03 22:23:01.621747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.621721    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"
Nov 03 22:23:01.621867 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.621757    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q" event={"ID":"715ce125-f53c-4b2e-848e-029866cbc5cf","Type":"ContainerDied","Data":"e7ac18da9fb3a39b5932e634d116b3eb400e300a0faca1843054c665868df542"}
Nov 03 22:23:01.623403 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.623384    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"
Nov 03 22:23:01.623507 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.623410    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw" event={"ID":"70773fa8-3c22-4f5f-8086-cc786e65c043","Type":"ContainerDied","Data":"3f3f399b2fdd648503ce438aa107b99733ffc8d5bc515812d3b1d394c927dd6a"}
Nov 03 22:23:01.630094 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.630076    5244 scope.go:122] "RemoveContainer" containerID="0de62eb2a43f31ac9cdaec25c7027cbaa10b801ccfbc6721510b2b9793ec379f"
Nov 03 22:23:01.638582 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.638566    5244 scope.go:122] "RemoveContainer" containerID="20bb461da74e86996bb5988499c52b6bd1fa9fe6c596e214e4e1a018e35226f5"
Nov 03 22:23:01.782135 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.782104    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"]
Nov 03 22:23:01.804320 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.804290    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/premium-w2-d71a74-kserve-7cd79dd77b-6l9bj"]
Nov 03 22:23:01.893565 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.893485    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"]
Nov 03 22:23:01.897778 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:01.897745    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/simulator-w2-d71a74-kserve-5f7b5f7cc7-v7tsw"]
Nov 03 22:23:02.005382 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:02.005343    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"]
Nov 03 22:23:02.008501 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:02.008474    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d71a74/distinct-w2-d71a74-kserve-97765c459-9rg4q"]
Nov 03 22:23:03.144079 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:03.144035    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" path="/var/lib/kubelet/pods/136a8937-a0c9-4e97-bf99-f4e708359238/volumes"
Nov 03 22:23:03.144652 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:03.144629    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" path="/var/lib/kubelet/pods/70773fa8-3c22-4f5f-8086-cc786e65c043/volumes"
Nov 03 22:23:03.145133 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:03.145114    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715ce125-f53c-4b2e-848e-029866cbc5cf" path="/var/lib/kubelet/pods/715ce125-f53c-4b2e-848e-029866cbc5cf/volumes"
Nov 03 22:23:03.220186 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:03.220150    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"]
Nov 03 22:23:03.220431 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:03.220406    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy" containerID="cri-o://7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec" gracePeriod=30
Nov 03 22:23:08.480678 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.480602    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 22:23:08.552215 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552179    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552235    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552271    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552306    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552429 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552381    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552439    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552471    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552529    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:08.552666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552532    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.552666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552625    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:08.552666 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552638    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token\") pod \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\" (UID: \"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215\") "
Nov 03 22:23:08.553014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552827    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:08.553014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552838    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:08.553014 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.552886    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:08.553438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.553041    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.553438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.553058    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.553438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.553067    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.553438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.553076    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.553438 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.553084    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.554947 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.554921    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:08.555065 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.554967    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:08.555242 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.555227    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:23:08.555295 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.555266    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79" pod "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" (UID: "c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215"). InnerVolumeSpecName "kube-api-access-rfg79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:08.654715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.654604    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfg79\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-kube-api-access-rfg79\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.654715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.654648    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.654715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.654663    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.654715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.654676    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:08.672366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.671835    5244 generic.go:356] "Generic (PLEG): container finished" podID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerID="7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec" exitCode=0
Nov 03 22:23:08.672366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.671922    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" event={"ID":"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215","Type":"ContainerDied","Data":"7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec"}
Nov 03 22:23:08.672366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.671974    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb" event={"ID":"c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215","Type":"ContainerDied","Data":"4a565123a1d2be5bed489abaed47b999d096f6a7f4be7d738f1434a99cd28cc4"}
Nov 03 22:23:08.672366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.672004    5244 scope.go:122] "RemoveContainer" containerID="7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec"
Nov 03 22:23:08.672366 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.672154    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"
Nov 03 22:23:08.686021 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.685993    5244 scope.go:122] "RemoveContainer" containerID="7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec"
Nov 03 22:23:08.686388 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:23:08.686361    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec\": container with ID starting with 7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec not found: ID does not exist" containerID="7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec"
Nov 03 22:23:08.686491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.686404    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec"} err="failed to get container status \"7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec\": rpc error: code = NotFound desc = could not find container \"7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec\": container with ID starting with 7a961eff4bae5b84bab5165d4b1db5343c7e82e81c5a9c2b5a9dbe104e0f69ec not found: ID does not exist"
Nov 03 22:23:08.835773 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.835672    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"]
Nov 03 22:23:08.839434 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:08.839405    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6de8e6-openshift-default-b48688d85-2qpsb"]
Nov 03 22:23:09.134749 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:09.134668    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" path="/var/lib/kubelet/pods/c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215/volumes"
Nov 03 22:23:15.057302 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:15.057260    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"]
Nov 03 22:23:15.057871 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:15.057481    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy" containerID="cri-o://79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b" gracePeriod=30
Nov 03 22:23:16.577551 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:16.577519    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"]
Nov 03 22:23:16.578026 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:16.577737    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" containerID="cri-o://888e0cd0c34d2bb848b45fcb0d65ff30311ee9f23ba1bd59f484ae8a231e226e" gracePeriod=30
Nov 03 22:23:20.316684 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.316660    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:23:20.473178 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473126    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473178 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473178    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473254    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473296    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473322    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473431    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473462    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473501    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473542 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473538    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket\") pod \"17edd6f0-d922-46ea-8ad7-428466cfdab9\" (UID: \"17edd6f0-d922-46ea-8ad7-428466cfdab9\") "
Nov 03 22:23:20.473903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473716    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:20.473903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473781    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:20.473903 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473849    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17edd6f0-d922-46ea-8ad7-428466cfdab9-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.474043 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473941    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:20.474043 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.473957    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:20.474625 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.474587    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:20.475866 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.475842    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:20.475973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.475876    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:20.475973 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.475914    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "kube-api-access-c4ffl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:20.476104 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.476089    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo" pod "17edd6f0-d922-46ea-8ad7-428466cfdab9" (UID: "17edd6f0-d922-46ea-8ad7-428466cfdab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:23:20.539565 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.539533    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"]
Nov 03 22:23:20.574483 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574439    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574483 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574472    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574483 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574486    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4ffl\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-kube-api-access-c4ffl\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574500    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574513    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574526    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574537    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.574789 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.574548    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17edd6f0-d922-46ea-8ad7-428466cfdab9-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:20.739794 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.739705    5244 generic.go:356] "Generic (PLEG): container finished" podID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerID="79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b" exitCode=0
Nov 03 22:23:20.739794 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.739757    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"
Nov 03 22:23:20.739989 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.739795    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" event={"ID":"17edd6f0-d922-46ea-8ad7-428466cfdab9","Type":"ContainerDied","Data":"79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b"}
Nov 03 22:23:20.739989 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.739853    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb" event={"ID":"17edd6f0-d922-46ea-8ad7-428466cfdab9","Type":"ContainerDied","Data":"7cd67e897598a8b6affd341683e43296973d8bd4fc536ad79397cd39406579a8"}
Nov 03 22:23:20.739989 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.739881    5244 scope.go:122] "RemoveContainer" containerID="79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b"
Nov 03 22:23:20.750816 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.750799    5244 scope.go:122] "RemoveContainer" containerID="cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"
Nov 03 22:23:20.759696 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.759671    5244 scope.go:122] "RemoveContainer" containerID="79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b"
Nov 03 22:23:20.760081 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:23:20.760060    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b\": container with ID starting with 79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b not found: ID does not exist" containerID="79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b"
Nov 03 22:23:20.760188 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.760092    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b"} err="failed to get container status \"79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b\": rpc error: code = NotFound desc = could not find container \"79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b\": container with ID starting with 79081d5d238513b83a93e8a1fbc9f7b3211f6fb3fe39fc8090d74e9a9102246b not found: ID does not exist"
Nov 03 22:23:20.760188 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.760112    5244 scope.go:122] "RemoveContainer" containerID="cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"
Nov 03 22:23:20.760464 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:23:20.760445    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0\": container with ID starting with cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0 not found: ID does not exist" containerID="cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"
Nov 03 22:23:20.760521 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.760470    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0"} err="failed to get container status \"cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0\": rpc error: code = NotFound desc = could not find container \"cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0\": container with ID starting with cb50f987170149ee6da90dc3237a42bb1260741cafc39543ccdb85a5a9ca6fb0 not found: ID does not exist"
Nov 03 22:23:20.812285 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.812237    5244 patch_prober.go:29] interesting pod/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 22:23:20.812491 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.812305    5244 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 22:23:20.902074 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.902034    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"]
Nov 03 22:23:20.905945 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:20.905915    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7fb477-openshift-default-57d94fd6f7-sshnb"]
Nov 03 22:23:21.137201 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.137086    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" path="/var/lib/kubelet/pods/17edd6f0-d922-46ea-8ad7-428466cfdab9/volumes"
Nov 03 22:23:21.752467 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.752432    5244 generic.go:356] "Generic (PLEG): container finished" podID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerID="888e0cd0c34d2bb848b45fcb0d65ff30311ee9f23ba1bd59f484ae8a231e226e" exitCode=0
Nov 03 22:23:21.752920 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.752499    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" event={"ID":"b87e33a0-e629-4b3b-a240-7d74779cbce8","Type":"ContainerDied","Data":"888e0cd0c34d2bb848b45fcb0d65ff30311ee9f23ba1bd59f484ae8a231e226e"}
Nov 03 22:23:21.824627 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.824603    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989598    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989648    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989685    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989744    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989780    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989845    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989871    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989951    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.989978    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token\") pod \"b87e33a0-e629-4b3b-a240-7d74779cbce8\" (UID: \"b87e33a0-e629-4b3b-a240-7d74779cbce8\") "
Nov 03 22:23:21.990459 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.990313    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:21.991214 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.990636    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:21.991853 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.991824    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:21.992000 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.991889    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:21.992071 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.992028    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 22:23:21.993544 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.993519    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "kube-api-access-z2b8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:21.993715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.993593    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:21.993715 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.993672    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 22:23:21.996788 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:21.996761    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy" pod "b87e33a0-e629-4b3b-a240-7d74779cbce8" (UID: "b87e33a0-e629-4b3b-a240-7d74779cbce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:22.091597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091567    5244 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091594    5244 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-credential-socket\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091597 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091606    5244 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87e33a0-e629-4b3b-a240-7d74779cbce8-istiod-ca-cert\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091614    5244 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-podinfo\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091623    5244 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-data\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091631    5244 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-token\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091641    5244 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-istio-envoy\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091648    5244 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87e33a0-e629-4b3b-a240-7d74779cbce8-workload-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.091897 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.091659    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2b8h\" (UniqueName: \"kubernetes.io/projected/b87e33a0-e629-4b3b-a240-7d74779cbce8-kube-api-access-z2b8h\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:22.435716 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:23:22.435680    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:23:22.760502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.760413    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj" event={"ID":"b87e33a0-e629-4b3b-a240-7d74779cbce8","Type":"ContainerDied","Data":"f487bf557ed9fd7c200c9fb0f8f7b1e636a42d90dae792e1bef8766bae70315f"}
Nov 03 22:23:22.760502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.760444    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"
Nov 03 22:23:22.760502 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.760472    5244 scope.go:122] "RemoveContainer" containerID="888e0cd0c34d2bb848b45fcb0d65ff30311ee9f23ba1bd59f484ae8a231e226e"
Nov 03 22:23:22.925645 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.925604    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"]
Nov 03 22:23:22.929674 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:22.929646    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7ee809-openshift-default-67875c59b9-ft8pj"]
Nov 03 22:23:23.134587 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:23.134503    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" path="/var/lib/kubelet/pods/b87e33a0-e629-4b3b-a240-7d74779cbce8/volumes"
Nov 03 22:23:30.963679 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:30.963646    5244 scope.go:122] "RemoveContainer" containerID="7902fa0a9bf85033510ddf234fa8523fdfbf3419defed916957c8f52a0647bef"
Nov 03 22:23:30.972683 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:30.972658    5244 scope.go:122] "RemoveContainer" containerID="abda7eb1bf2773715e8b86bd470fcc8f958d9cc34b058b578cdc50e1efc4d650"
Nov 03 22:23:32.128164 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:32.128125    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:23:35.559877 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.559826    5244 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main" containerID="cri-o://2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033" gracePeriod=45
Nov 03 22:23:35.796843 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.796818    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 22:23:35.828652 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.828566    5244 generic.go:356] "Generic (PLEG): container finished" podID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerID="2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033" exitCode=0
Nov 03 22:23:35.828652 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.828622    5244 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"
Nov 03 22:23:35.828828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.828660    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" event={"ID":"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3","Type":"ContainerDied","Data":"2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033"}
Nov 03 22:23:35.828828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.828712    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4" event={"ID":"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3","Type":"ContainerDied","Data":"c301a91eb4f35bf8df68186d31e79e35e07eea87e47fbd9f7bdedb35ead28652"}
Nov 03 22:23:35.828828 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.828729    5244 scope.go:122] "RemoveContainer" containerID="2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033"
Nov 03 22:23:35.837534 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.837508    5244 scope.go:122] "RemoveContainer" containerID="2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033"
Nov 03 22:23:35.837890 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:23:35.837866    5244 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033\": container with ID starting with 2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033 not found: ID does not exist" containerID="2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033"
Nov 03 22:23:35.837976 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.837913    5244 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033"} err="failed to get container status \"2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033\": rpc error: code = NotFound desc = could not find container \"2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033\": container with ID starting with 2c33630e89ac2921fc80f0ae6609fc1c2c139234659b3c41bd2fa8691fe8d033 not found: ID does not exist"
Nov 03 22:23:35.906414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906375    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs\") pod \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") "
Nov 03 22:23:35.906414 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906414    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg\" (UniqueName: \"kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg\") pod \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") "
Nov 03 22:23:35.906656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906441    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home\") pod \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") "
Nov 03 22:23:35.906656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906483    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache\") pod \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") "
Nov 03 22:23:35.906656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906507    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm\") pod \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") "
Nov 03 22:23:35.906656 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906579    5244 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir\") pod \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\" (UID: \"c9ed677e-1d69-432d-9d1b-2b6b7c9216f3\") "
Nov 03 22:23:35.906885 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906765    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home" pod "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" (UID: "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.906885 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906779    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache" pod "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" (UID: "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.906959 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906886    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir" pod "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" (UID: "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.906959 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906907    5244 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-home\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.906959 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.906926    5244 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-model-cache\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:35.908995 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.908968    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm" pod "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" (UID: "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 22:23:35.908995 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.908985    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs" pod "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" (UID: "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 22:23:35.909137 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:35.908984    5244 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg" pod "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" (UID: "c9ed677e-1d69-432d-9d1b-2b6b7c9216f3"). InnerVolumeSpecName "kube-api-access-2xklg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 22:23:36.007940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:36.007907    5244 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tls-certs\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:36.007940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:36.007936    5244 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xklg\" (UniqueName: \"kubernetes.io/projected/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-kube-api-access-2xklg\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:36.007940 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:36.007949    5244 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-dshm\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:36.008198 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:36.007958    5244 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3-tmp-dir\") on node \"ip-10-0-129-65.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 22:23:36.247099 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:36.247031    5244 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"]
Nov 03 22:23:36.252373 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:36.252347    5244 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-da5f0b/simulator-w3-da5f0b-kserve-549bcd58b4-vrnz4"]
Nov 03 22:23:37.134986 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:37.134955    5244 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" path="/var/lib/kubelet/pods/c9ed677e-1d69-432d-9d1b-2b6b7c9216f3/volumes"
Nov 03 22:23:49.745939 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:23:49.745902    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:23:52.435401 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:23:52.435360    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:24:03.319000 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:03.318962    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qzfp9_8c917408-568c-4cff-9395-3f6263463100/limitador/0.log"
Nov 03 22:24:05.072570 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:05.072538    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7t5j_27601318-46e6-42fd-9ba7-a4f6d32538fc/manager/0.log"
Nov 03 22:24:06.061061 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:06.061019    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cdbc4ddd8-6drd4_b2a36d9c-2c6a-475a-92d5-3caef091151b/manager/0.log"
Nov 03 22:24:07.004952 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:07.004909    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qzfp9"]
Nov 03 22:24:19.143941 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:19.143906    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7t5j_27601318-46e6-42fd-9ba7-a4f6d32538fc/manager/0.log"
Nov 03 22:24:19.848017 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:19.847980    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cdbc4ddd8-6drd4_b2a36d9c-2c6a-475a-92d5-3caef091151b/manager/0.log"
Nov 03 22:24:21.443675 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:21.443641    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c4b54ff46-bp45f_5168928d-67f0-459f-9658-4e280b7446a9/maas-api/0.log"
Nov 03 22:24:22.427563 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:22.427533    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3f0cbed5-xnqv5_83c83d91-edc1-4c87-bd80-9918a9ebe7f5/revoke-keys/0.log"
Nov 03 22:24:22.436211 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:24:22.436178    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:24:23.543680 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:23.543643    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-78b527eb-fqdg6_f8956965-8df9-45f5-b88a-3ff623369d4d/revoke-keys/0.log"
Nov 03 22:24:23.771999 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:23.771970    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e4063bf5-gpkbq_4bfa5197-b69d-411d-a8b7-01d4ebad5407/revoke-keys/0.log"
Nov 03 22:24:23.994640 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:23.994610    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-msqdw_db456a3d-1c18-4567-978f-0b653438b0d0/postgres/0.log"
Nov 03 22:24:27.551260 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:27.551226    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qzfp9_8c917408-568c-4cff-9395-3f6263463100/limitador/0.log"
Nov 03 22:24:28.835039 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:28.835007    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5nzpr_09ac9c37-d9fb-4e07-b242-f2edc1413f68/discovery/0.log"
Nov 03 22:24:29.419805 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:29.419772    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545b54f677-5rl57_0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b/kube-auth-proxy/0.log"
Nov 03 22:24:31.039029 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:31.038996    5244 scope.go:122] "RemoveContainer" containerID="469a7eea0080bdc3fca8b532cfe815a30de53b845e45fe27a13ad4cc894c9253"
Nov 03 22:24:31.047630 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:31.047607    5244 scope.go:122] "RemoveContainer" containerID="46e5e5e17fd5a38afb6aa2141cbfafb03bd81c58a2a2dff18bed817ef6e208c3"
Nov 03 22:24:52.436055 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:24:52.436014    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:24:53.319682 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:53.319655    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s77gw_8541995b-77c8-4973-9736-f750edf753cb/global-pull-secret-syncer/0.log"
Nov 03 22:24:56.881236 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:56.881194    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h4zdr_5eeb32df-a293-4351-8743-a3b12b67dd87/konnectivity-agent/0.log"
Nov 03 22:24:58.248774 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:24:58.248734    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-65.us-west-1.compute.internal_29b854c94d02e7fd862ea1754302117f/haproxy/0.log"
Nov 03 22:25:06.783960 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:06.783921    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7t5j_27601318-46e6-42fd-9ba7-a4f6d32538fc/manager/0.log"
Nov 03 22:25:07.210099 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:07.210057    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cdbc4ddd8-6drd4_b2a36d9c-2c6a-475a-92d5-3caef091151b/manager/0.log"
Nov 03 22:25:12.666739 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:12.666702    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-l7t5j_27601318-46e6-42fd-9ba7-a4f6d32538fc/manager/0.log"
Nov 03 22:25:13.121732 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:13.121692    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cdbc4ddd8-6drd4_b2a36d9c-2c6a-475a-92d5-3caef091151b/manager/0.log"
Nov 03 22:25:19.302252 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:19.302213    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5nzpr_09ac9c37-d9fb-4e07-b242-f2edc1413f68/discovery/0.log"
Nov 03 22:25:19.409839 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:19.409811    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545b54f677-5rl57_0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b/kube-auth-proxy/0.log"
Nov 03 22:25:21.982747 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.982710    5244 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"]
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983050    5244 cpu_manager.go:414] "containerMap: removing container" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983061    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983073    5244 cpu_manager.go:414] "containerMap: removing container" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983079    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983087    5244 cpu_manager.go:414] "containerMap: removing container" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983092    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983101    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983106    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983111    5244 cpu_manager.go:414] "containerMap: removing container" podUID="f8956965-8df9-45f5-b88a-3ff623369d4d" containerName="revoke-keys"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983115    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8956965-8df9-45f5-b88a-3ff623369d4d" containerName="revoke-keys"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983129    5244 cpu_manager.go:414] "containerMap: removing container" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983134    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983145    5244 cpu_manager.go:414] "containerMap: removing container" podUID="715ce125-f53c-4b2e-848e-029866cbc5cf" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983150    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715ce125-f53c-4b2e-848e-029866cbc5cf" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983161    5244 cpu_manager.go:414] "containerMap: removing container" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983165    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983172    5244 cpu_manager.go:414] "containerMap: removing container" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy"
Nov 03 22:25:21.983194 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983177    5244 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983226    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="715ce125-f53c-4b2e-848e-029866cbc5cf" containerName="main"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983235    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b36ef1-e007-42ab-ac1c-ad5e6a2cd215" containerName="istio-proxy"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983240    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8956965-8df9-45f5-b88a-3ff623369d4d" containerName="revoke-keys"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983248    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ed677e-1d69-432d-9d1b-2b6b7c9216f3" containerName="main"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983252    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87e33a0-e629-4b3b-a240-7d74779cbce8" containerName="istio-proxy"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983258    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="70773fa8-3c22-4f5f-8086-cc786e65c043" containerName="main"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983263    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="136a8937-a0c9-4e97-bf99-f4e708359238" containerName="main"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983268    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy"
Nov 03 22:25:21.983814 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.983273    5244 memory_manager.go:365] "RemoveStaleState removing state" podUID="17edd6f0-d922-46ea-8ad7-428466cfdab9" containerName="istio-proxy"
Nov 03 22:25:21.986279 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:21.986263    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.038408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.038376    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ph99f\"/\"openshift-service-ca.crt\""
Nov 03 22:25:22.038408 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.038377    5244 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ph99f\"/\"kube-root-ca.crt\""
Nov 03 22:25:22.067816 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.067785    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-podres\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.067918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.067821    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-sys\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.067918 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.067850    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-lib-modules\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.067990 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.067964    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-proc\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.068026 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.068003    5244 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kns\" (UniqueName: \"kubernetes.io/projected/8e1489db-7d62-4dee-9519-dfba01807f54-kube-api-access-k8kns\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.087895 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.087866    5244 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ph99f\"/\"default-dockercfg-vjzx4\""
Nov 03 22:25:22.089071 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.089051    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"]
Nov 03 22:25:22.168845 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168813    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-sys\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168857    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-lib-modules\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168884    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-proc\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168907    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kns\" (UniqueName: \"kubernetes.io/projected/8e1489db-7d62-4dee-9519-dfba01807f54-kube-api-access-k8kns\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168954    5244 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-podres\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168975    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-lib-modules\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168977    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-sys\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.168977    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-proc\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.169054 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.169028    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8e1489db-7d62-4dee-9519-dfba01807f54-podres\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.225653 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.225618    5244 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kns\" (UniqueName: \"kubernetes.io/projected/8e1489db-7d62-4dee-9519-dfba01807f54-kube-api-access-k8kns\") pod \"perf-node-gather-daemonset-dn7b6\" (UID: \"8e1489db-7d62-4dee-9519-dfba01807f54\") " pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.298381 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.298285    5244 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:22.435704 ip-10-0-129-65 kubenswrapper[5244]: E1103 22:25:22.435667    5244 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvcz7" containerName="extractor"
Nov 03 22:25:22.667980 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:22.667905    5244 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"]
Nov 03 22:25:22.671084 ip-10-0-129-65 kubenswrapper[5244]: W1103 22:25:22.671053    5244 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8e1489db_7d62_4dee_9519_dfba01807f54.slice/crio-7009db64d4a4defc2ecfc69e62210958050f99de2e442ec65cb20940fd9f52a9 WatchSource:0}: Error finding container 7009db64d4a4defc2ecfc69e62210958050f99de2e442ec65cb20940fd9f52a9: Status 404 returned error can't find the container with id 7009db64d4a4defc2ecfc69e62210958050f99de2e442ec65cb20940fd9f52a9
Nov 03 22:25:23.355143 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:23.355099    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6" event={"ID":"8e1489db-7d62-4dee-9519-dfba01807f54","Type":"ContainerStarted","Data":"52dc17ed96deac34752a6ef509579aacf0eeda2d97c280ed5e7f34ea79c14931"}
Nov 03 22:25:23.355603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:23.355182    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:23.355603 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:23.355208    5244 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6" event={"ID":"8e1489db-7d62-4dee-9519-dfba01807f54","Type":"ContainerStarted","Data":"7009db64d4a4defc2ecfc69e62210958050f99de2e442ec65cb20940fd9f52a9"}
Nov 03 22:25:23.468607 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:23.468534    5244 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6" podStartSLOduration=2.468514057 podStartE2EDuration="2.468514057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:23.467821993 +0000 UTC m=+4194.451818552" watchObservedRunningTime="2026-11-03 22:25:23.468514057 +0000 UTC m=+4194.452510617"
Nov 03 22:25:27.750489 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:27.750460    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qzfp9_8c917408-568c-4cff-9395-3f6263463100/limitador/0.log"
Nov 03 22:25:29.370842 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:29.370815    5244 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ph99f/perf-node-gather-daemonset-dn7b6"
Nov 03 22:25:31.073374 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:31.073323    5244 scope.go:122] "RemoveContainer" containerID="f6af507f3411561a2ddb71a7222eb64a12386ae47ba793c05e1aa2b6317862aa"
Nov 03 22:25:31.085296 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:31.085271    5244 scope.go:122] "RemoveContainer" containerID="449cfa24f1fa03b6cdc988bb983055c7fa8738dba3b6cec588bb6577641f769a"
Nov 03 22:25:45.822831 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:45.822794    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5nzpr_09ac9c37-d9fb-4e07-b242-f2edc1413f68/discovery/0.log"
Nov 03 22:25:45.931119 ip-10-0-129-65 kubenswrapper[5244]: I1103 22:25:45.931082    5244 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-545b54f677-5rl57_0e5f8c7f-c09b-4b32-bf94-c6343bb4ad3b/kube-auth-proxy/0.log"